文件列表:

.adr-dir (28, 2023-12-14)
.bin.sample/ (0, 2023-12-14)
.bin.sample/restart (30, 2023-12-14)
.bin.sample/restart-and-build (64, 2023-12-14)
.bin.sample/start (182, 2023-12-14)
.cfignore (4, 2023-12-14)
.circleci/ (0, 2023-12-14)
.circleci/config.yml (3366, 2023-12-14)
.coveragerc (629, 2023-12-14)
.dockerignore (32, 2023-12-14)
.editorconfig (198, 2023-12-14)
.pre-commit-config.yaml (94, 2023-12-14)
.pylintrc (591, 2023-12-14)
CHANGELOG.md (317813, 2023-12-14)
CODEOWNERS (19, 2023-12-14)
Dockerfile (483, 2023-12-14)
LICENCE (1091, 2023-12-14)
Makefile (2204, 2023-12-14)
Procfile (174, 2023-12-14)
app.py (535, 2023-12-14)
codecov.yml (91, 2023-12-14)
config/ (0, 2023-12-14)
config/__init__.py (0, 2023-12-14)
config/api_docs_urls.py (1546, 2023-12-14)
config/api_urls.py (4204, 2023-12-14)
config/settings/ (0, 2023-12-14)
config/settings/__init__.py (0, 2023-12-14)
config/settings/common.py (25089, 2023-12-14)
config/settings/common_logging.py (1185, 2023-12-14)
config/settings/dev.py (194, 2023-12-14)
config/settings/local.py (1677, 2023-12-14)
config/settings/mi/ (0, 2023-12-14)
config/settings/production.py (45, 2023-12-14)
... ...

# Data Hub API [](https://circleci.com/gh/uktrade/data-hub-api) [](https://codecov.io/gh/uktrade/data-hub-api) Data Hub API provides an API into Data Hub for Data Hub clients. Using Data Hub API you can search for entities and manage companies, contacts and interactions. More guides can be found in the [docs](./docs/) folder. To instantiate the development environment, please follow one of the three following sets of instructions depending on your use case: ## Installation with Docker This project uses Docker compose to setup and run all the necessary components. The docker-compose.yml file provided is meant to be used for running tests and development. **Note for Mac Users:** By default, docker on Mac will restrict itself to using just 2GB of memory. This [should be increased](https://docs.docker.com/docker-for-mac/#resources) to at least 4GB to avoid running in to unexpected problems. 1. Clone the repository: ```shell git clone https://github.com/uktrade/data-hub-api cd data-hub-api ``` 2. Create `.env` files from `sample.env` ```shell cp sample.env .env cp config/settings/sample.env config/settings/.env ``` If you're working with data-hub-frontend and mock-sso, `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` should be the same as `MOCK_SSO_EMAIL_USER_ID` in mock-sso environment definition in [data-hub-frontend/docker-compose.frontend.yml](https://github.com/uktrade/data-hub-frontend/blob/main/docker-compose.frontend.yml) 3. Build and run the necessary containers for the required environment: ```shell docker-compose up ``` or ```shell make start-dev ``` - It will take time for the API container to come up - it will run migrations on both DBs, load initial data, sync opensearch etc. Watch along in the api container's logs. - **NOTE:** If you are using a linux system, the opensearch container may not come up successfully (`data-hub-api_es_1`) - it might be perpetually restarting. If the logs for that container mention something like `max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]`, you will need to run the following on your host machine: ```shell sudo sysctl -w vm.max_map_count=262144 ``` and append/modify the `vm.max_map_count` setting in `/etc/sysctl.conf` (so that this setting persists after restart): ```shell vm.max_map_count=262144 ``` For more information, [see the opensearch docs on vm.max_map_count](https://opensearch.org/docs/latest/opensearch/install/important-settings/). 4. Optionally, you may want to run a local copy of the data hub frontend. By default, you can run both the API and the frontend under one docker-compose project. [See the instructions in the frontend readme to set it up](https://github.com/uktrade/data-hub-frontend/#running-project-within-docker). Alternatively use the `make` command documented below if you also want to bring up `dnb-service` ### Installation with docker of data-hub-api, data-hub-frontend and dnb-service on same network There is now a `make` command to bring up the three environments on a single docker network, allowing the services to talk to each other effortlessly 1. Clone the repositories ```shell git clone https://github.com/uktrade/data-hub-api git clone https://github.com/uktrade/data-hub-frontend git clone https://github.com/uktrade/dnb-service cd data-hub-api ``` 2. Create `.env` files from `sample.env` ```shell cp sample.env .env cp config/settings/sample.env config/settings/.env ``` Ensure `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` is the same as `MOCK_SSO_EMAIL_USER_ID` in mock-sso environment definition in [data-hub-frontend/docker-compose.frontend.yml](https://github.com/uktrade/data-hub-frontend/blob/main/docker-compose.frontend.yml) and `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` the same as `DJANGO_SUPERUSER_EMAIL` in data-hub-api .env file otherwise the user may not exist You will also need to add `DNB_SERVICE_BASE_URL` and `DNB_SERVICE_TOKEN` to dnb-service/.env in order to use DnB functionality. These credentials can be found in Vault 3. Run make command ```shell make start-frontend-api-dnb ``` - It will take time for the API container to come up - it will run migrations on both DBs, load initial data, sync opensearch etc. Watch along in the api container's logs. - **NOTE:** If you are using a linux system, the opensearch container may not come up successfully (`data-hub-api_es_1`) - it might be perpetually restarting. If the logs for that container mention something like `max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]`, you will need to run the following on your host machine: ```shell sudo sysctl -w vm.max_map_count=262144 ``` and append/modify the `vm.max_map_count` setting in `/etc/sysctl.conf` (so that this setting persists after restart): ```shell vm.max_map_count=262144 ``` For more information, [see the opensearch docs on vm.max_map_count](https://opensearch.org/docs/latest/opensearch/install/important-settings/). 4. If you want to stop all the services, run the following make command ```shell make stop-frontend-api-dnb ``` ## Native installation (without Docker) Dependencies: - Python 3.10.x - PostgreSQL 12 - redis 6.x - OpenSearch 1.x 1. Clone the repository: ```shell git clone https://github.com/uktrade/data-hub-api cd data-hub-api ``` 2. Install Python 3.10. [See this guide](https://docs.python-guide.org/starting/installation/) for detailed instructions for different platforms. 3. Install system dependencies: On Ubuntu: ```shell sudo apt install build-essential libpq-dev python3.10-dev python3.10-venv ``` On macOS: ```shell brew install libpq ``` 4. Install _postgres_, if not done already, as this is required by **psycopg2** in the requirements below On Ubuntu: ```shell sudo apt install postgresql postgresql-contrib ``` On macOS: ```shell brew install postgresql ``` 5. Create and activate the virtualenv: ```shell python3.10 -m venv env source env/bin/activate pip install -U pip or formally to make sure you have the same version as what is used for cloudfoundry, use buildpack to install the same version e.g. https://github.com/cloudfoundry/python-buildpack/releases e.g. 22.1.2 python -m pip install pip==22.1.2 ``` 6. Install the dependencies: ```shell pip install -r requirements-dev.txt ``` 7. Create an `.env` settings file (it’s gitignored by default): ```shell cp config/settings/sample.env config/settings/.env ``` 8. Set `DOCKER_DEV=False` and `LOCAL_DEV=True` in `.env` 9. Make sure you have OpenSearch running locally. If you don't, you can run one in Docker: ```shell docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "plugins.security.disabled=true" opensearchproject/opensearch:1.2.4 ``` 10. Make sure you have redis running locally and that the REDIS_BASE_URL in your `.env` is up-to-date. 11. Populate the databases and initialise OpenSearch: ```shell ./manage.py migrate ./manage.py migrate_search ./manage.py loadinitialmetadata ./manage.py createinitialrevisions ``` 12. Optionally, you can load some test data: ```shell ./manage.py loaddata fixtures/test_data.yaml ``` Note that this will queue RQ tasks to index the created records in OpenSearch, and hence the loaded records won‘t be returned by search endpoints until RQ is started and the queued tasks have run. 13. Create a superuser: ```shell ./manage.py createsuperuser ``` (You can enter any valid email address as the username and SSO email user ID.) 14. Start the server: ```shell ./manage.py runserver ``` 15. Start RQ (Redis Queue): ```shell python rq/rq-worker.py ``` Note that in production the cron-scheduler:1/1, short-running-working:4/4, long-running-worker:4/4 are run in separate instances . ## API documentation Automatically-generated API documentation is served at `/docs` (requires admin site credentials). ## Local development If using Docker, prefix these commands with `docker-compose run api`. To run the tests: ```shell ./tests.sh ``` To run the tests in parallel, pass `-n ` to `./tests.sh`. For example, for four processes: ```shell ./tests.sh -n 4 ``` By default all tests are executed. To skip tests that should not run under certain conditions ie. are dependent on external services, use a marker `excluded` or a marker that start with `excluded_`. ```python import pytest @pytest.mark.excluded class TestClass: def test_1(self): ... @pytest.mark.excluded def test_1(): ... @pytest.mark.excluded_dependent_on_redis def test_2(): ... ``` To exclude such tests from running use one of the options below: ```shell pytest --skip-excluded pytest -m excluded_dependent_on_redis pytest -m "excluded_x or excluded_y" pytest -m "not (excluded_x or excluded_y)" ``` **NOTE:** When testing, using the `--reuse-db` can speed up test runs by excluding migrations To run the linter: ```shell flake8 ``` There is also a pre-commit hook for Flake8. To install this, run: ```shell make setup-flake8-hook ``` ### Obtaining an API access token You can obtain an access token for local development work in one of two ways: - by running `./manage.py add_access_token ` with the SSO email user ID of an existing adviser (run `./manage.py add_access_token --help` for a list of options) - using the form on `http://localhost:8000/admin/add-access-token/` (If you’re using Docker, an access token will be created automatically if certain environment variables are set. See `sample.env` for more details.) This access token can be used with most endpoints by setting an `Authorization` header value of `Bearer `. Note that machine-to-machine endpoints (such as those under `/v4/metadata/`) instead use Hawk authentication and request signing. ## Granting access to the front end The [internal front end](https://github.com/uktrade/data-hub-frontend) uses single sign-on. You should configure the API as follows to use with the front end: - `SSO_ENABLED`: `True` - `STAFF_SSO_BASE_URL`: URL of a [Staff SSO](https://github.com/uktrade/staff-sso) or [Mock SSO](https://github.com/uktrade/mock-sso) instance. This should be the same server the front end is configured to use. - `STAFF_SSO_AUTH_TOKEN`: Access token for Staff SSO. ## Granting access to machine-to-machine clients Pure machine-to-machine clients use Hawk authentication with separate credentials for each client. There are separate views for such clients as these views don’t expect `request.user` to be set. Hawk credentials for each client are defined in settings below and each client is assigned scopes in `config/settings/common.py`. These scopes define which views each client can access. ## Deployment Data Hub API can run on any Heroku-style platform. Configuration is performed via the following environment variables: | Variable name | Required | Description | | --------------------------------------------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ACTIVITY_STREAM_ACCESS_KEY_ID` | No | A non-secret access key ID, corresponding to `ACTIVITY_STREAM_SECRET_ACCESS_KEY`. The holder of the secret key can access the activity stream endpoint by Hawk authentication. | | `ACTIVITY_STREAM_SECRET_ACCESS_KEY` | If `ACTIVITY_STREAM_ACCESS_KEY_ID` is set | A secret key, corresponding to `ACTIVITY_STREAM_ACCESS_KEY_ID`. The holder of this key can access the activity stream endpoint by Hawk authentication. | | `ACTIVITY_STREAM_OUTGOING_URL` | No | The URL used to read from activity stream | | `ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID` | No | A non-secret access key ID, corresponding to `ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY`. This is used when reading from the activity stream at `ACTIVITY_STREAM_OUTGOING_URL`. | | `ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY` | No | A secret key, corresponding to `ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID`. This is used when reading from the activity stream at `ACTIVITY_STREAM_OUTGOING_URL`. | | `ADMIN_OAUTH2_ENABLED` | Yes | Enables Django Admin SSO login when is True. | | `ADMIN_OAUTH2_BASE_URL` | If `ADMIN_OAUTH2_ENABLED` is set | A base URL of OAuth provider. | | `ADMIN_OAUTH2_TOKEN_FETCH_PATH` | If `ADMIN_OAUTH2_ENABLED` is set | OAuth fetch token path for Django Admin SSO login. | | `ADMIN_OAUTH2_USER_PROFILE_PATH` | If `ADMIN_OAUTH2_ENABLED` is set | OAuth user profile path for Django Admin SSO login. | | `ADMIN_OAUTH2_AUTH_PATH` | If `ADMIN_OAUTH2_ENABLED` is set | OAuth auth path for Django Admin SSO login. | | `ADMIN_OAUTH2_CLIENT_ID` | If `ADMIN_OAUTH2_ENABLED` is set | OAuth client ID for Django Admin SSO login. | | `ADMIN_OAUTH2_CLIENT_SECRET` | If `ADMIN_OAUTH2_ENABLED` is set | OAuth client secret for Django Admin SSO login. | | `AV_V2_SERVICE_URL` | Yes | URL for ClamAV V2 service. If not configured, virus scanning will fail. | | `AWS_ACCESS_KEY_ID` | No | Used as part of [boto3 auto-configuration](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials). | | `AWS_DEFAULT_REGION` | No | [Default region used by boto3.](http://boto3.readthedocs.io/en/latest/guide/configuration.html#environment-variable-configuration) | | `AWS_SECRET_ACCESS_KEY` | No | Used as part of [boto3 auto-configuration](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials). | | `COMPANY_MATCHING_SERVICE_BASE_URL` | No | The base url of the company matching service (default=None). | | `COMPANY_MATCHING_HAWK_ID` | No | The hawk id to use when making a request to the company matching service (default=None). | | `COMPANY_MATCHING_HAWK_KEY` | No | The hawk key to use when making a request to the company matching service (default=None). | | `CONSENT_SERVICE_BASE_URL` | No | The base url of the consent service, to post email consent preferences to (default=None). | | `CONSENT_SERVICE_HAWK_ID` | No | The hawk id to use when making a request to the consent service (default=None). | | `CONSENT_SERVICE_HAWK_KEY` | No | The hawk key to use when making a request to the consent service (default=None). | | `CSRF_COOKIE_HTTPONLY` | No | Whether to use HttpOnly flag on the CSRF cookie (default=False). | | `CSRF_COOKIE_SECURE` | No | Whether to use a secure cookie for the CSRF cookie (default=False). | | `DATA_FLOW_API_ACCESS_KEY_ID` | No | A non-secret access key ID, corresponding to `DATA_FLOW_API_SECRET_ACCESS_KEY`. The holder of the secret key can access the om ... ...

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：