dlts-enm:DLTS增强型网络专着

  • B4_578860
    了解作者
  • 36.4MB
    文件大小
  • zip
    文件格式
  • 0
    收藏次数
  • VIP专享
    资源类型
  • 0
    下载次数
  • 2022-06-15 00:52
    上传日期
enm-DLTS增强网络专着 CLI应用程序,用于生成增强型网络专着(ENM)静态网站和Solr索引。 概述 enm是一个CLI应用程序,用于执行各种后端ENM功能: 创建静态页面 关于 家 创建浏览主题列表:例如 创建主题页面:例如 加载enm-pages Solr索引 自动创建缓存的数据文件,可以在后续作业中代替Postgres数据库使用: enm程序尚未生产。 完成了足够的开发,以创建一个初始的稳定且正确的演示站点。 ENM数据当前处于冻结状态,目前尚无活动计划来添加或更改数据。 入门 先决条件 (建议至少1.10) (如果使用deploy shell脚本) 数据库 的Postgres数据库应该已经使用正确的用户凭据在devdb1.dlib.nyu.edu上建立了。 有关详细信息,请参见 。 安装与设定 使用go get安装: go get github.com/nyulibr
dlts-enm-develop.zip
内容介绍
# enm - DLTS Enhanced Network Monographs CLI application for generating the Enhanced Networked Monographs (ENM) static website and Solr index. ## Overview `enm` is a CLI application for performing various backend ENM functions: * Create static pages * About * Home * Create browse topics lists: e.g. [featured topics](http://enm.dlib.nyu.edu/browse-topics-lists/enm-picks.html) * Create topic pages: e.g. [culture -- popular](http://enm.dlib.nyu.edu/topic-pages/00/00/00/78/0000007805.html) * Load `enm-pages` Solr index * Automatically create cached data files which can be used in place of the Postgres database in subsequent jobs: [nyudlts/enm\-cache](https://github.com/nyudlts/enm-cache) The `enm` program has not been productionized. Enough development was done to create an initial stable and correct demo site. ENM data is currently frozen and there are no active plans to add or change data at this time. ## Getting Started ### Prerequisities * [Go](https://golang.org/) (at least 1.10 recommended) * [dep](https://golang.github.io/dep/) * [Expect](https://core.tcl.tk/expect/index) (if using the deploy shell scripts) #### Database The Postgres database from [TCT](https://github.com/NYULibraries/dlts-enm-tct-backend) should already be set up on devdb1.dlib.nyu.edu with the correct user credentials. See https://jira.nyu.edu/jira/browse/NYUP-437 for details. ### Installation and setup Installation using `go get`: ```shell go get github.com/nyulibraries/dlts-enm cd dlts-enm/ git remote rm origin git remote add origin git@github.com:NYULibraries/dlts-enm.git git fetch --all dep ensure go build mv dlts-enm enm ``` Installation using `git clone`: ```shell mkdir -p $GOPATH/src/github.com/NYULibraries/ cd $GOPATH/src/github.com/NYULibraries/ git clone git@github.com:NYULibraries/dlts-enm.git cd dlts-enm/ dep ensure go build mv dlts-enm enm ``` #### Set environment variables Set environment variables for database access: ```shell export ENM_POSTGRES_DATABASE=enm export ENM_POSTGRES_DATABASE_HOSTNAME=127.0.0.1 export ENM_POSTGRES_DATABASE_USERNAME=enm_readonly export ENM_POSTGRES_DATABASE_PASSWORD=[password for devdb1.dlib.nyu.edu:enm database for user enm_readonly] ``` Note that we use 127.0.0.1 even though the database is remote because we'll need to access the remote Postgres server through an SSH tunnel through the bastion host. These environment variables must be set before running any `enm` command, even the ones that do not technically need database access. Failure to do so will cause a `panic`. Set location of the cache using ENM_CACHE: ```shell script export ENM_CACHE=$HOME/enm-cache/ ``` This environment variable is optional. If it is set and the path that is pointed to does not already exist, `enm` will create it, along with any needed intermediate directories. If `ENM_CACHE` is not set, the location of the cache defaults to `/tmp/enm-cache/`. #### Set up SSH tunnel to devdb1 Set up SSH tunneling to port 5432 on Postgres host devdb1.dlib.nyu.edu through bastion host by running this command in a separate terminal: ```shell ssh -N -L 5432:devdb1.dlib.nyu.edu:5432 [USERNAME]@b.dlib.nyu.edu ``` This will allow remote database access via local port 5432. A less verbose command can be run if the following is set up in `.ssh/config`: ```shell Host devdb1 Hostname devdb1.dlib.nyu.edu ProxyCommand ssh bastion -W %h:%p User [USERNAME] ... Host bastion Hostname b.dlib.nyu.edu User [USERNAME] ``` In a separate terminal, run this command: ```shell ssh -N -L 5432:devdb1:5432 bastion ``` ### Usage #### Deploy site There is a deploy script that can generate the full website and copy it to the dev, stage, and prod web servers. The easiest way to use it is to run the wrapper script that will prompt for information and then run the deploy script with the proper options and credentials. The `rsync` commands are run using [`expect`](https://core.tcl.tk/expect/index) to automatically enter the user credentials for the bastion and web server hosts. * `bin/deploy-site_expect.sh dev` * `bin/deploy-site_expect.sh stage` * `bin/deploy-site_expect.sh prod` The deploy script runs all the `sitegen` commands detailed below with destination set to directories in `dist/`. See [examples](#examples) for a full demonstration of how to use this wrapper script. Note that the deploy script does static page generation and copying to server only, it does not perform Solr indexing. #### Get general help `./enm help` #### Get help on a command `./enm help sitegen` #### Get help on a subcommmand `./enm help sitegen browsetopicslists` #### Create browse topics lists using Postgres database `./enm sitegen browsetopicslists --destination=[DESTINATION]` #### Create browse topics lists using cache files `./enm sitegen browsetopicslists --destination=[DESTINATION] --source=cache` #### Create page About, Home, etc. `./enm sitegen sitepages --destination=[DESTINATION]` #### Create all topic pages using Postgres database This automatically creates cache files in /tmp/enm-cache/sitegen-topicpages/ that can be used as the data source for subsequent topic pages generation runs: `./enm sitegen topicpages --source=database --destination=[DESTINATION]` #### Create all topic pages using cache files in /tmp/enm-cache/sitegen-topicpages/ `./enm sitegen topicpages --source=cache --destination=[DESTINATION]` #### Create topic pages for 2 topics only using Postgres database `./enm sitegen topicpages --source=database --destination=[DESTINATION] [TOPIC ID 1] [TOPIC ID 2]` #### Create topic pages for 2 topics only using cache files `./enm sitegen topicpages --source=cache --destination=[DESTINATION] [TOPIC ID 1] [TOPIC ID 2]` #### Load enm-pages Solr index using Postgres database `./enm solr load --server=[SOLR SERVER] --port 8983` #### Load enm-pages Solr index using cache files `./enm solr load --server=[SOLR SERVER] --source=cache --port 8983` ### Examples #### Create the production website from latest cache files cloned to $HOME/enm-cache/ and deploy to the dev web server In the example below, it is assumed that the `dlts-enm` repo is located at `$GOPATH/src/github.com/nyulibraries/dlts-enm/`, and the `https://github.com/nyudlts/enm-cache` repo has already been cloned to $HOME. ```shell $ export ENM_CACHE=$HOME/enm-cache/ $ bin/deploy-site_expect.sh dev Do complete regeneration of the site before copying to server? [y/n] y Use the cache for regenerating the site? [y/n] y Username for b.dlib.nyu.edu and devweb1.dlib.nyu.edu: someuser Password for b.dlib.nyu.edu and devweb1.dlib.nyu.edu: spawn /Users/someuser/Documents/programming/go/gopath/src/github.com/nyulibraries/dlts-enm/bin/deploy-site.sh -c -g -u arjanik dev Generating site pages... Generating browse topics lists... Generating topic pages... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [...SNIPPED...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ someuser@b.dlib.nyu.edu's password: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [...SNIPPED...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ someuser@devweb1.dlib.nyu.edu's password: building file list ... done about.html sent 127 bytes received 66 bytes 77.20 bytes/sec total size is 2.46K speedup is 12.76 rsync #1 completed successfully. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED ~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [...SNIPPED...] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
评论
    相关推荐
    • SpringBoot + solr
      学习solr写的demo https://github.com/99246255/SpringBoot-Solr
    • dineshjagai.github.io:文件夹
      dineshjagai.github.io:文件夹
    • solr-ocrhighlighting:直接在Solr中突出显示各种OCR格式
      使用此Solr插件,您可以将单词级OCR文本放入一个或多个文档的字段中,然后在查询时获取带有文本及其在页面上位置的结构化突出显示数据。 所有这些都无需在索引本身中存储OCR数据,而是重新使用磁盘上现有的OCR文件。...
    • rpgorganizertutorial.github.io
      rpgorganizertutorial.github.io
    • islandora_solution_pack_meme:查询所有模因的solr
      Islandora Meme解决方案包 这个虚拟模块gives是为科罗拉多州的Islandora Camp创建的,目的是为您提供学习如何在Islandora中查询solr的框架。 引导您的VM ... 接下来,从github的该模块的“ content
    • ik-analyzer-solr7
      ik-analyzer-solr7,支持 solr7,内含 jar 包和使用文档,亲测可用
    • rspec-solr:宝石
      RSpec Solr 一个提供用于与Solr响应对象一起使用的RSpec自定义匹配器的宝石。 安装 将此行添加到您的应用程序的Gemfile中: gem 'rspec-solr' 然后执行: $ bundle 或将其自己安装为: $ gem install rspec-solr...
    • plugin-solr:dotCMS系统的SOLR插件
      dotCMS SOLR插件 dotCMS系统的SOLR插件可用于将内容和所有相关字段发布到SOLR安装中以进行索引。 该插件包含一个Portlet,该Portlet显示要编制索引的内容的队列,以及可以在自定义工作流中使用的工作流Actionlet,...
    • solr-cool.github.io:Solr软件包目录和保护区
      该网站使用构建,并托管在Github Pages上。 使用官方Docker映像启动本地Jekyll实例,并将浏览器指向 。 docker run --rm \ --volume="$PWD:/srv/jekyll" \ -p "4000:4000" \ -it jekyll/jekyll:3.8 \ jekyll ...
    • solr服务器.zip
      solr,检索工具,此工具通常用在全文检索,效率比数据库自带的模糊查询快,solr检索效率高,占用资源低