dlts-enm：DLTS增强型网络专着

# enm - DLTS Enhanced Network Monographs CLI application for generating the Enhanced Networked Monographs (ENM) static website and Solr index. ## Overview `enm` is a CLI application for performing various backend ENM functions: * Create static pages * About * Home * Create browse topics lists: e.g. [featured topics](http://enm.dlib.nyu.edu/browse-topics-lists/enm-picks.html) * Create topic pages: e.g. [culture -- popular](http://enm.dlib.nyu.edu/topic-pages/00/00/00/78/0000007805.html) * Load `enm-pages` Solr index * Automatically create cached data files which can be used in place of the Postgres database in subsequent jobs: [nyudlts/enm\-cache](https://github.com/nyudlts/enm-cache) The `enm` program has not been productionized. Enough development was done to create an initial stable and correct demo site. ENM data is currently frozen and there are no active plans to add or change data at this time. ## Getting Started ### Prerequisities * [Go](https://golang.org/) (at least 1.10 recommended) * [dep](https://golang.github.io/dep/) * [Expect](https://core.tcl.tk/expect/index) (if using the deploy shell scripts) #### Database The Postgres database from [TCT](https://github.com/NYULibraries/dlts-enm-tct-backend) should already be set up on devdb1.dlib.nyu.edu with the correct user credentials. See https://jira.nyu.edu/jira/browse/NYUP-437 for details. ### Installation and setup Installation using `go get`: ```shell go get github.com/nyulibraries/dlts-enm cd dlts-enm/ git remote rm origin git remote add origin git@github.com:NYULibraries/dlts-enm.git git fetch --all dep ensure go build mv dlts-enm enm ``` Installation using `git clone`: ```shell mkdir -p $GOPATH/src/github.com/NYULibraries/ cd $GOPATH/src/github.com/NYULibraries/ git clone git@github.com:NYULibraries/dlts-enm.git cd dlts-enm/ dep ensure go build mv dlts-enm enm ``` #### Set environment variables Set environment variables for database access: ```shell export ENM_POSTGRES_DATABASE=enm export ENM_POSTGRES_DATABASE_HOSTNAME=127.0.0.1 export ENM_POSTGRES_DATABASE_USERNAME=enm_readonly export ENM_POSTGRES_DATABASE_PASSWORD=[password for devdb1.dlib.nyu.edu:enm database for user enm_readonly] ``` Note that we use 127.0.0.1 even though the database is remote because we'll need to access the remote Postgres server through an SSH tunnel through the bastion host. These environment variables must be set before running any `enm` command, even the ones that do not technically need database access. Failure to do so will cause a `panic`. Set location of the cache using ENM_CACHE: ```shell script export ENM_CACHE=$HOME/enm-cache/ ``` This environment variable is optional. If it is set and the path that is pointed to does not already exist, `enm` will create it, along with any needed intermediate directories. If `ENM_CACHE` is not set, the location of the cache defaults to `/tmp/enm-cache/`. #### Set up SSH tunnel to devdb1 Set up SSH tunneling to port 5432 on Postgres host devdb1.dlib.nyu.edu through bastion host by running this command in a separate terminal: ```shell ssh -N -L 5432:devdb1.dlib.nyu.edu:5432 [USERNAME]@b.dlib.nyu.edu ``` This will allow remote database access via local port 5432. A less verbose command can be run if the following is set up in `.ssh/config`: ```shell Host devdb1 Hostname devdb1.dlib.nyu.edu ProxyCommand ssh bastion -W %h:%p User [USERNAME] ... Host bastion Hostname b.dlib.nyu.edu User [USERNAME] ``` In a separate terminal, run this command: ```shell ssh -N -L 5432:devdb1:5432 bastion ``` ### Usage #### Deploy site There is a deploy script that can generate the full website and copy it to the dev, stage, and prod web servers. The easiest way to use it is to run the wrapper script that will prompt for information and then run the deploy script with the proper options and credentials. The `rsync` commands are run using [`expect`](https://core.tcl.tk/expect/index) to automatically enter the user credentials for the bastion and web server hosts. * `bin/deploy-site_expect.sh dev` * `bin/deploy-site_expect.sh stage` * `bin/deploy-site_expect.sh prod` The deploy script runs all the `sitegen` commands detailed below with destination set to directories in `dist/`. See [examples](#examples) for a full demonstration of how to use this wrapper script. Note that the deploy script does static page generation and copying to server only, it does not perform Solr indexing. #### Get general help `./enm help` #### Get help on a command `./enm help sitegen` #### Get help on a subcommmand `./enm help sitegen browsetopicslists` #### Create browse topics lists using Postgres database `./enm sitegen browsetopicslists --destination=[DESTINATION]` #### Create browse topics lists using cache files `./enm sitegen browsetopicslists --destination=[DESTINATION] --source=cache` #### Create page About, Home, etc. `./enm sitegen sitepages --destination=[DESTINATION]` #### Create all topic pages using Postgres database This automatically creates cache files in /tmp/enm-cache/sitegen-topicpages/ that can be used as the data source for subsequent topic pages generation runs: `./enm sitegen topicpages --source=database --destination=[DESTINATION]` #### Create all topic pages using cache files in /tmp/enm-cache/sitegen-topicpages/ `./enm sitegen topicpages --source=cache --destination=[DESTINATION]` #### Create topic pages for 2 topics only using Postgres database `./enm sitegen topicpages --source=database --destination=[DESTINATION] [TOPIC ID 1] [TOPIC ID 2]` #### Create topic pages for 2 topics only using cache files `./enm sitegen topicpages --source=cache --destination=[DESTINATION] [TOPIC ID 1] [TOPIC ID 2]` #### Load enm-pages Solr index using Postgres database `./enm solr load --server=[SOLR SERVER] --port 8983` #### Load enm-pages Solr index using cache files `./enm solr load --server=[SOLR SERVER] --source=cache --port 8983` ### Examples #### Create the production website from latest cache files cloned to $HOME/enm-cache/ and deploy to the dev web server In the example below, it is assumed that the `dlts-enm` repo is located at `$GOPATH/src/github.com/nyulibraries/dlts-enm/`, and the `https://github.com/nyudlts/enm-cache` repo has already been cloned to $HOME. ```shell $ export ENM_CACHE=$HOME/enm-cache/ $ bin/deploy-site_expect.sh dev Do complete regeneration of the site before copying to server? [y/n] y Use the cache for regenerating the site? [y/n] y Username for b.dlib.nyu.edu and devweb1.dlib.nyu.edu: someuser Password for b.dlib.nyu.edu and devweb1.dlib.nyu.edu: spawn /Users/someuser/Documents/programming/go/gopath/src/github.com/nyulibraries/dlts-enm/bin/deploy-site.sh -c -g -u arjanik dev Generating site pages... Generating browse topics lists... Generating topic pages... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [...SNIPPED...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ someuser@b.dlib.nyu.edu's password: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [...SNIPPED...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ someuser@devweb1.dlib.nyu.edu's password: building file list ... done about.html sent 127 bytes received 66 bytes 77.20 bytes/sec total size is 2.46K speedup is 12.76 rsync #1 completed successfully. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [...SNIPPED...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~