Building Hong Kong Automated Teller Machine (ATM) Locator

 ## Project Synopsis There are three ATM networks in Hong Kong, which are HSBC, Hang Seng Bank and JETCO respectively. ATM data will be retrieved via API Portal from HSBC, Hang Seng Bank and APIX. However, the interoperability is frustrating as the API is implemented at different standard levels. Hence, this project aims to centralise Hong Kong ATM data in a well-defined yet standardised format and display in a web portal for public use. Hong Kong Monetary Authority (HKMA) has published [Open API Framework for the Hong Kong Banking Sector](https://www.hkma.gov.hk/media/eng/doc/key-information/press-release/2018/20180718e5a2.pdf), mentioning that no standardised open API functions will be provided at the first release. Paragraph #20 > Throughout the discussion and consultation period, the HKMA recognises the industry’s desire to see a common set of Open APIs for better interoperability. However, a number of international banks operating in Hong Kong have already implemented their group standard for implementing Open APIs at global or regional levels, and have demonstrated elsewhere that requiring banks to adhere to a prescribed set of standardised Open API functions is challenging. Paragraph #21 > Some opinions from the technology sector also indicate that it would be more desirable for banks to quickly offer Open APIs than to wait for standardised Open APIs that would take time to emerge. Furthermore, it is believed that once an ecosystem has been developed and becomes mature, convergence to standardised Open APIs will likely occur in response to the needs of the market. ## Data Retrieval At the beginning, I needed to register an account in order to retrieve the data. However, it required me to provide BR number and fill in a lot of stuff. The process was a bit tedious and time-confusing. At the end, I only received the following API info. - [x] Retrieving ATM data from HSBC and Hang Seng Banks from API Portals - [ ] Retrieving ATM data from APIX - [ ] Bank of China (Hong Kong) Limited - [x] Bank of Communications (Hong Kong) Limited - [x] Bank of Communications Co., Ltd. - [x] China CITIC Bank International Limited - [x] China Construction Bank (Asia) Corporation Limited - [x] China Merchants Bank Hong Kong Branch - [ ] Chiyu Banking Corporation Limited - [x] Chong Hing Bank Limited - [ ] Citibank (Hong Kong) Limited - [x] CMB Wing Lung Bank Limited - [ ] Dah Sing Bank, Limited - [ ] DBS Bank (Hong Kong) Limited - [x] Fubon Bank (Hong Kong) Limited - [x] Industrial and Commercial Bank of China (Asia) Limited - [ ] Nanyang Commercial Bank Limited - [x] OCBC Wing Hang Bank Limited - [ ] Public Bank (Hong Kong) Limited - [ ] Shanghai Commercial Bank Limited - [ ] Standard Chartered Bank (Hong Kong) Limited - [x] The Bank of East Asia, Limited As I haven't received others' reply and all the required data is not complete, this project is currently archived. ## Hackathon I presented this project in one of the g0vhk hackathons and I was looking for different contributors. Technical Contributors: - Responsible for implementing features and fixing reported bugs - Preferably with experience with knowledge in Open API and Web Development Non-technical Contributors: - Responsible for updating documentation - Good command of written in English and Chinese UI/UX Contributors: - Responsible for refining UI/UX - Knowledge of UI/UX principles and techniques At the end, I only met several people but their opinions were valuable. ## Frontend The frontend part is built with - ReactJS - Library for building user interfaces - Material UI - React components that implement Google's Material Design - React-Leaflet - Interactive OSM map The listing page shows the selected ATM network in a list view, sorted by the distance. If Location in your browswer is turned off, the default location (22.308, 114.1716) will be used. You may also see this location in my another project [geodesy](https://pub.dev/packages/geodesy/versions/0.3.0). If you google it, it shows Eaton HK where we had hackathons by [g0v.hk](https://www.facebook.com/g0vhk.io) and [The Loop](https://www.facebook.com/groups/loop.dev).  By clicking the button on the top-right side, it switches to a map view.  Clicking the corresponding item navigates to a detail page showing the basic info and the map.  The UI is relatively simple. Below shows how to prepare the data and transform it to the desired format. ## Scrapers At an early stage of development, ATM data were fetched from corresponding bank websites. See [here](https://github.com/wingkwong/hk-atm-locator/tree/master/archive/scrapers) for the scrapers. Here's some sample data. ``` code,districtName,districtCode,name,latitude,longitude,address,service _central_western_district,Central & Western District,hong_kong_district,Central District Branch,22.280249,114.160161,"2A Des Voeux Road Central, Hong Kong",ATM_RMB _central_western_district,Central & Western District,hong_kong_district,Bonham Road Branch,22.2844809,114.1409265,"63 Bonham Road, Hong Kong",ATM_RMB _central_western_district,Central & Western District,hong_kong_district,Shek Tong Tsui Branch,22.2862301,114.1345169,"534 Queen's Road West, Shek Tong Tsui, Hong Kong",ATM_RMB _central_western_district,Central & Western District,hong_kong_district,Kennedy Town Branch,22.2835392,114.129382,"Harbour View Garden, 2-2F Catchick Street, Kennedy Town, Hong Kong","ATM_RMB,CASH_DEPOSIT_DUAL_CURRENCY" _central_western_district,Central & Western District,hong_kong_district,Connaught Road Central Branch,22.282915,114.157785,"13-14 Connaught Road Central, Hong Kong",ATM_RMB _central_western_district,Central & Western District,hong_kong_district,Caine Road Branch,22.2808553,114.1526749,"57 Caine Road, Hong Kong",ATM_RMB ``` Later on, those data were retrieved from HSBC, Hang Seng API Portals and APIX instead. However, as I couldn't get all the ATM data. Brian Leung shared an idea that scraping the data from JETCO because it provides most of the data I missed. Hence, I wrote a simple Python program to scrape the data. Here's an example. ``` {"xml_msg": {"@query": "region=2,area=1,district=10,transaction=0", "supp_cb": null, "atms": {"atm": [{"@id": "627", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "Bank of China (Hong Kong) Limited", "addr": "Podium Of Haking Wong Bldg., University Of Hong Kong, Pok Fu Lam, Hong Kong.", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS"]}, "currencies": {"currency": ["HKD", "RMB"]}, "latitude": "22.282941420489305", "longitude": "114.13639426231384"}, {"@id": "628", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "Bank of China (Hong Kong) Limited", "addr": "Shop 510, Chi Fu Landmark, Pok Fu Lam, Hong Kong", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS", "FISC"]}, "currencies": {"currency": ["HKD", "RMB"]}, "latitude": "22.257842107377694", "longitude": "114.13875728845596"}, {"@id": "215658", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "Bank of China (Hong Kong) Limited", "addr": "Shop No.22 at G/F, Wah Fu (I) Shopping Centre, 23 Wah Fu Road, Pokfulam, Hong Kong.", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS", "FISC"]}, "currencies": {"currency": ["HKD", "RMB"]}, "latitude": "22.2501635", "longitude": "114.13746650000007"}, {"@id": "228553", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "Bank of China (Hong Kong) Limited", "addr": "G/F, Wing A, Main Hospital Building, Queen Mary Hospital, 102 Pokfulam Road, HK", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS", "FISC"]}, "currencies": {"currency": ["HKD", "RMB"]}, "latitude": "22.270076", "longitude": "114.131498"}, {"@id": "291814", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "The Bank of East Asia, Limited", "addr": "Shop P0030, G/F, Centennial Campus, The University of Hong Kong", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Interbank Transfer", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS", "FISC", "Diners Club", "Discover"]}, "currencies": {"currency": ["HKD", "RMB"]}, "latitude": "22.2833372", "longitude": "114.13428799999997"}, {"@id": "291813", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "The Bank of East Asia, Limited", "addr": "Shop P0030, G/F, Centennial Campus, The University of Hong Kong", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Interbank Transfer", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS", "FISC", "Diners Club", "Discover"]}, "currencies": {"currency": "HKD"}, "latitude": "22.2833372", "longitude": "114.13428799999997"}, {"@id": "162131", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "The Bank of East Asia, Limited", "addr": "Outdoor Area of 2/F Chong Yuet Ming Amenities Centre, Main Campus, The University of Hong Kong", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "MPF", "Interbank Transfer", "Bill Payment, Credit Card Payment and Charity Donation", "PLUS", "JCB", "CUP", "CIRRUS", "Diners Club", "FISC", "Discover"]}, "currencies": {"currency": "HKD"}, "latitude": "22.28272588988215", "longitude": "114.13904817699892"}, {"@id": "171376", "region": "Hong Kong", "area": "Hong Kong", "district": "Pokfulam", "ob_name": "Industrial and Commercial Bank of China (Asia) Limited", "addr": "HKU ATM2", "supp_tran": {"tran_name": ["Cash Withdrawal, Fund Transfer, Balance Enquiry, PIN Change and other basic services", "Interbank Transfer", "Bill Payment, Credit Card Payment and Charity Donation", "JCB", "CUP", "CIRRUS"]}, "currencies": {"currency": ["HKD", "RMB"]}, "latitude": "22.283728578971324", "longitude": "114.13659663795931"}]}}} ``` Even the data may not be up-to-dated, it is still better than none. ## Data Transformation I wrote several programs to transform, enrich, and manipulate data from API Portals and produce data in a well-defined yet standardised format. The transformers includes the following scripts: ### prepare_data.js The script is used to fetch data from API Portals. An example to prepare HSBC data: ``` const prepareHsbcData = async (outputFile) => { info('Start to prepare the hsbc data'); const res = await request.getAsync({ url: configs.HSBC_API_ENDPOINT, headers: { ...header, ClientID: configs.HSBC_CLIENT_ID, ClientSecret: configs.HSBC_CLIIENT_SECRET, }, }); remind(`Successfully fetched the hsbc data.Size: ${res.body.length} `); fs.writeFileSync(outputFile, res.body, 'utf8'); remind(`Successfully store the data to ${outputFile} `); }; ``` ### process_data.js This script is used to transform or enrich the data. For example, data like operating hour displaying as ``24-hours`` needs to be converted back to a standarised format. ``` const createGenericOpeningHours = (openTime, closeTime) => { const WEEK_DAYS = [ 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday' ]; return WEEK_DAYS.map(weekday => ({ OpenDayDescription: weekday, OpenTime: openTime, CloseTime: closeTime })); } ``` ### generate_checksum.js As we fetch the data periodically, data may be not updated. Hence, this file is to generate the checksum of the processed files. If the checksum is same, it won't copy the data from ``transformer/processed/.json`` to ``web/src/data`` A simple checksum generator using nodeJS built-in crypto. ``` const generateChecksum = (data, md5Path) => { const checksum = crypto .createHash('md5') .update(data, 'utf8') .digest('hex'); if(shouldWriteChecksum(md5Path, checksum)) { fs.writeFileSync(md5Path, checksum); remind(`Finished generating checksum file at ${md5Path}`); } else { fs.unlinkSync(md5Path); } } const shouldWriteChecksum = (md5Path, checksum) => { if(!fs.existsSync(md5Path)) return true; const checksumInmd5 = fs.readFileSync(md5Path); return checksum === checksumInmd5 ? false : true; } ``` ### process.js It is a CLI Program using ``commander``. It is the entry point to call other functions. ``` const program = require('commander'); program .version('0.1.0'); /** * Get the address */ program .command('process-address ') .description('fetch and get the address and save to file') .action(processAddress); /** * Get the data from the API Portal */ program .command('prepare ') .description('Get the data from the bank') .action(prepareData); /** * Process the raw data */ program .command('process ') .description('Process the prepared data and output it') .action(processData); /** * Generate the checksum */ program .command('process-checksum ') .description('Check and write checksum of the processed data') .action(generateChecksum); program.parse(process.argv); // If no arguments we should output the help if (!program.args.length) program.help(); ``` It also uses ``hk-address-parser-lib``, which happens to be one of the projects that I've worked on, to enrich latitude and longitude based on Address Line. ``` async function parseAddress(atm) { const addressLine = atm.ATMAddress.AddressLine.join(' '); const records = await AddressParser.parse(addressLine); if (records.length > 0) { const { lat, lng } = records[0].coordinate(); atm.ATMAddress.LatitudeDescription = lat + ''; // eslint-disable-line atm.ATMAddress.LongitudeDescription = lng + ''; // eslint-disable-line } } ``` The general flow is shown as below - Pipe processing Hang Seng data ```bash ./src/process.js prepare hang_seng unprocessed/hang_seng.json && \ ./src/process.js process hang_seng unprocessed/hang_seng.json processing/hang_seng.json && \ ./src/process.js process-address hang_seng processing/hang_seng.json processed/hang_seng.json ``` - Pipe processing HSBC data ```bash ./src/process.js prepare hsbc unprocessed/hsbc.json && \ ./src/process.js process hsbc unprocessed/hsbc.json processing/hsbc.json && \ ./src/process.js process-address hsbc processing/hsbc.json processed/hsbc.json ``` - Pipe processing JETCO data ```bash ./src/process.js process jetco unprocessed/jetco/en/ processed/jetco_en.json && \ ./src/process.js process jetco unprocessed/jetco/tc/ processed/jetco_tc.json ``` - Generate Checksum files after pipe processing each network ```bash ./src/process.js process-checksum hang_seng processed/hang_seng.json checksum/hang_seng.md5 && \ ./src/process.js process-checksum hsbc processed/hsbc.json checksum/hsbc.md5 && \ ./src/process.js process-checksum jetco processed/jetco_en.json checksum/jetco_en.md5 && \ ./src/process.js process-checksum jetco processed/jetco_tc.json checksum/jetco_tc.md5 ``` If there is no change in checksum file, the data file will not be copied and committed. ## CI/CD I hosted the website on Github Pages only. Thanks to Nandi Wong for setting up Travis CI, the abovementioned process is now automated. Basically it includes jobs to install dependencies and execute corresponding scripts to pull the latest data. ## Conclusion Even though this project could not make it at the end, I learnt quite a lot and met a lot of brilliant people. Thanks all the contributors throughout these months. The project can be found in my Github. Here's the [link](https://github.com/wingkwong/hk-atm-locator).

Designing Cloud Native Apps in GCP

## Selecting the appropriate Cloud Service Model - Infrastructure-as-a-service (IaaS) model - large degree of fliexibility in implementation - requries a significant amount of labor - Software-as-a-service (SaaS) model - higher velocity for delivery of services while maintaining a fair amount of flexibility - at the expense of flexibility - Iaas -> CaaS -> PaaS -> Faas -> Saas (from low fliexibility to high velocity) ## Protability and Design Consideration - Protable Languages - Platform considerations - Platform specific designs are targeted for a specific environment - Vendor lock in ## Evaluating Different Service Technologies - Google App Engine - you want to focus on writing code, never wants to touch a server, cluster or infrastructure - you want to build a highly reliable and scalable serving app or component without doing it all yourself - you value developer velocity over infrastructure control - Minimize operational overhead - Google Kubernetes Engine - you want to increase velocity and improve operability dramatically by separating the app from the OS - you need a secure, scalable way to manage containers in production - you don't have dependencies on a specific operating system - Google Compute Engine - you need complete control over your infrastructure and direct access to high-performance hardware such as GPUs and local SSDs - you need to make OS-level changes, such as providing your own network or graphic drivers, to squeeze out the last drop of performance - You want to move your application from your own colo or datacenter to the cloud without rewriting it - You need to run a software package that can't easily be containerized or your want to use existing VM images ## Operating System Considerations - CentOS - Container-optimized OS from Google - CoreOS - Debian - Red Hat Enterprise Linux (RHEL) - SUSE Enterprise Linux - SLEX for SAP - Ubuntu - Windows Server ## Location of Your Service Components - to cut down on latency and provide better services to your end users ## Microservice Architectures - Separated into independent constituent parts, with each part having its own realm of responisbility - Refactored from monolithic apps that have very tight coupling to a micro-service based architectures - Advantages: - the code base becomes more modular and easier to manage - it becomes much easier to reuse services for other applications - it is much easier to scale and tune individules services ## Defining Key Structures - avoid monotonically increasing keys - instead, migrate to keys that use random numbers, such as UUID ## Session Management - keep a session cache - Cloud Spanner - a limit of 10K sessions per database per node - a client can delete a session - The Cloud Spanner database service can delete a session when the session is idle for more than 1 hour ## API Management Consideration - Apps should be designed to have loosely coupled components - Pub/Sub model enables event-driven architectures & asyn parallel processing - Pulisher---(publish event)---> Event Channel --(Fire Event)---> Subscriber - Pulisher---(publish event)---> Event Channel <--(Subscribe)---- Subscriber ## Health Checks - Cloud Load Balancing - Interal Load Balancing - TCP Proxy Load Balancing - SSL Proxy Load Balancing - HTTP(s) Load Balancing - Stackdriver Monitoring --- uptime check ---> storage/database/network - Example: ``` gcloud compute health-checks create [PROTOCOl] [HEALTH_CHECK_NAME] \ --description=[DESCRIPTION] \ --check-interval=[CHECK_INTERVAL] \ --timeout=[TIMEOUT] \ --healthy-threshold=[HEALTHY_THRESHOLD] \ --unhealthy-threshold=[UNHEALTHY_THRESHOLD] \ ...additional flags ```

Auto-upgrading nodes in GCP Containers

## Checking the state of auto-upgrade for an existing node pool ``` gcloud container node-pools describe node-pool-name \ --cluster cluster-name \ --zone compute-zone ``` ## Enabling node auto-upgrades for an existing node pool ``` gcloud container node-pools update node-pool-name --cluster cluster-name \ --zone compute-zone --enable-autoupgrade ``` ## Disabling node auto-upgrades for an existing node pool where: ``` gcloud container node-pools update node-pool-name --cluster cluster-name \ --zone compute-zone --no-enable-autoupgrade ``` - node-pool-name is the name of the node pool. - cluster-name is the name of the cluster that contains the node pool. - compute-zone is the zone for the cluster.

Azure Synapse Analytics 101

- formerly Azure SQL Data Warehouse (SQL DW) - Massively Parallel Processing (MPP) Data Warehouse - Connection Security - Firewall rules are used by both the server and the database to reject connection attempts from IP addresses that haven't been explicitly whitelisted. - Authentication - SQL pool currently supports SQL Server Authentication with a username and password, and with Azure Active Directory. - Authorization - Authorization privileges are determined by role memberships and permissions. Authorization privileges are determined by role memberships and permissions. - Data Encryption - protects against the threat of malicious activity by encrypting and decrypting your data at rest. Associated backups and transaction log files are encrypted without requiring any changes to your applications when encrypting your database. - Advanced Data Security - provides a set of advanced SQL security capabilities, including data discovery & classification, vulnerability assessment, and Advanced Threat Protection. ## Transparent Data Encryption (TDE) - It helps protect against the threat of malicious activity by encrypting and decrypting your data at rest. When you encrypt your database, associated backups and transaction log files are encrypted without requiring any changes to your applications. TDE encrypts the storage of an entire database by using a symmetric key called the database encryption key. ## Granular access controls - Granular Permissions let you control which operations you can do on individual columns, tables, views, schemas, procedures, and other objects in the database. Use granular permissions to have the most control and grant the minimum permissions necessary. - Database roles other than db_datareader and db_datawriter can be used to create more powerful application user accounts or less powerful management accounts. The built-in fixed database roles provide an easy way to grant permissions, but can result in granting more permissions than are necessary. - Stored procedures can be used to limit the actions that can be taken on the database. ## Service Type - Compute Optimized Gen1 - Compute Optimized Gen2 ## Scalability - Linear Scale on data warehouse unit ## Backup - Use data warehouse snapshot to create a restore point

Deleting objects in GCP using Lifecycle Policy File

Example: Delete the object after 31 days. ```json { "rule": [ { "action": {"type": "Delete"}, "condition": {"age": 31} } ] } ``` Set the policy ``` gsutil lifecycle set life.json gs://$BUCKET_NAME_1 ``` Get the policy ``` gsutil lifecycle get gs://$BUCKET_NAME_1 ```