Crimegraph:犯罪网络中的实时视觉分析应用程序,用于链接预测和链接检测。 第五届IEEE物联网与云的未来国际会议(FiC

  • Z8_811565
    了解作者
  • 216.3KB
    文件大小
  • zip
    文件格式
  • 0
    收藏次数
  • VIP专享
    资源类型
  • 0
    下载次数
  • 2022-06-15 07:45
    上传日期
犯罪记录 利用数据流处理进行犯罪网络分析 发现犯罪网络中感兴趣的模式的能力可以支持和减轻安全和执法机构的调查任务。 作为犯罪网络,社交网络的一种特殊情况,我们可以适当地重用大多数最新技术来发现利益模式,即隐藏和潜在的联系。 但是,在时间敏感的场景中,例如涉及犯罪行为的场景,以(近)实时方式发现模式的能力可能是最重要的。 在本文中,我们研究了在不断发展的犯罪网络上用于链接检测和预测的模式识别。 作为犯罪网络的动态社交图谱,我们利用流处理方法在数据生成后立即提取感兴趣的信息。 为此,我们还提出了三个新的相似性社交网络指标,专门针对犯罪链接检测和预测而量身定制。 然后,我们依靠Apache Flink框架开发一个灵活的数据流处理应用程序; 该解决方案允许部署和评估新提议的指标以及文献中现有的指标。 实验结果表明,我们提出的新指标可以达到83%的检测准确度和82%的预测准确度,因此与最新的指标具有竞争力。 要求 要在本地执行该应用,您需要在系统上安装以下软件: Java Maven Flink Neo4J 卡夫卡 Kafka客户端(Python) 要将应用程序部署到Digital Oc
crimegraph-master.zip
内容介绍
# CRIMEGRAPH *Criminal networks analysis leveraging data stream processing* The ability to discover patterns of interest in criminal networks can support and ease the investigation tasks by security and law enforcement agencies. Being the criminal network a special case of social network, we can properly reuse most of the state-of-the-art techniques to discover patterns of interests, i.e., hidden and potential links. Nevertheless, in time-sensible scenarios, like the one involving criminal actions, the ability to discover patterns in a (near) real-time manner can be of primary importance. In this paper, we investigate the identification of patterns for link detection and prediction on an evolving criminal network. Being the criminal network a dynamic social graph, we exploit a stream processing approach to extract information of interest as soon as data is generated. To this end, we also propose three new similarity social network metrics, specifically tailored for criminal link detection and prediction. Then, we develop a flexible data stream processing application, relying on the framework Apache Flink; this solution allows to deploy and evaluate the newly proposed metrics as well as the one existing in literature. The experimental results show that the new metrics we propose can reach up to 83% accuracy in detection and 82% accuracy in prediction, resulting competitive with the state of the art metrics. ## Requirements To execute the app locally you need the following to be installed to your system: * Java * Maven * Flink * Neo4J * Kafka * Kafka client (Python) To deploy the app to a Digital Ocean droplet, you need the following to be installed on your system: * Vagrant * Vagrant plugin for Digital Ocean * Ansible ## Setup First you need to create the Kafka topic `main-topic`: $kafka-home> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic main-topic Test the topic creation: $kafka-home> bin/kafka-topics.sh --list --zookeeper localhost:2181 To test message publishing: $kafka-home> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic main-topic $kafka-home> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic main-topic ## Build The app building is provided by Apache Maven. To build the app you need to run $crimegraph> mvn clean package -Pbuild-jar If you want to skip tests, add the profile `skip-tests`. Copy the target `target/crimegraph-1.0.jar` into `data/flink/flink_master/crimegraph/crimegraph-1.0.jar`. ## Deploy Provision EC2 instances: $crimegraph> vagrant up Every time you want to test some changes, run: $crimegraph>mvn clean package -Pbuild-jar $crimegraph>vagrant rsync $crimegraph>vagrant provision To destroy all the EC2 instances: $crimegraph>vagrant destroy ## Usage $crimegraph> vagrant ssh kafka_master -c "sudo systemctl restart kafka" $crimegraph-monitor> vagrant ssh -c "sudo crimegraph-monitor check_kafka --kafkaBroker [EC2_KAFKA_MASTER]:9092 --kafkaTopic main-topic" $crimegraph-monitor> vagrant ssh -c "sudo crimegraph-monitor check_kafka --kafkaBroker [EC2_KAFKA_MASTER]:9092 --kafkaTopic main-topic" where *[EC2_KAFKA_MASTER]* is the public address of the EC2 instance named *kafka_master*. You need to run *crimegraph-monitor check_kafka* twice, because the first time creates the topic, while the second one tests that everything is ok. If *crimegraph check_kafka* returns *true*, everything is ok, and you can proceed to start Crimegraph. $crimegraph> vagrant ssh flink_master -c "sudo flink run /vagrant/target/crimegraph-1.0.jar --config /vagrant/data/instance/crimegraph/config.yaml" Now you can monitor the running application, visiting `[EC2_FLINK_MASTER]:8081` and `[EC2_NEO4J_MASTER]:7474`. It is time to submit some data! First of all, list all the available datasets: $crimegraph-monitor> vagrant ssh -c "sudo ls /vagrant/data/datasets" Generate trainset and testset for detection: $crimegraph-monitor> vagrant ssh -c "sudo crimegraph-monitor traintest_detection --dataset /vagrant/data/datasets/[DATASET] --trainset /vagrant/data/datasets/[DATASET]\_train_detection.data --testset /vagrant/data/datasets/[DATASET]\_test_detection.data --testRatio [RATIO]" where *[DATASET]* is the chosen dataset and *[RATIO]* is the test ratio, that is a number in (0.0,1.0) indicating the percentage of original dataset that will be deleted (a typical value is 0.10). Verify that datasets have been created: $crimegraph-monitor> vagrant ssh -c "sudo ls /vagrant/data/datasets" Now publish the trainset against Crimegraph: $crimegraph-monitor> vagrant ssh -c "sudo crimegraph-monitor publish --dataset /vagrant/data/datasets/[DATASET]\_train_detection.data --kafkaBroker [EC2_KAFKA_MASTER]:9092 --kafkaTopic main-topic" where *[EC2_KAFKA_MASTER]* is the public address of the EC2 instance named *kafka_master*. Check the publication status: $crimegraph-monitor> vagrant ssh -c "sudo crimegraph-monitor check_dataset_db --dataset /vagrant/data/datasets/[DATASET]\_train_detection.data --neo4jHostname bolt://[EC2_NEO4J_MASTER] --neo4jUsername neo4j --neo4jPassword password --timeout 10" where *[EC2_NEO4J_MASTER]* is the public address of the EC2 instance named *neo4j_master*. Now evaluate AUC for detection: $crimegraph-monitor> vagrant ssh -c "sudo crimegraph-monitor auc_detection --neo4jHostname bolt://[EC2_NEO4J_MASTER]:7687 --neo4jUsername neo4j --neo4jPassword password --dataset /vagrant/data/datasets/[DATASET] --trainset /vagrant/data/datasets/[DATASET]\_train_detection.data --testset /vagrant/data/datasets/[DATASET]\_test_detection.data --output /vagrant/data/[DATASET]\_auc_detection.out" ## Authors Giacomo Marciani, [gmarciani@acm.org](mailto:gmarciani@acm.org) Michele Porretta, [mporretta@acm.org](mailto:mporretta@acm.org) ## References Giacomo Marciani, Michele Porretta. 2017. *Crimegraph*. Series. Organization, Country [Read here](https://gmarciani.com) ## License The project is released under the [MIT License](https://opensource.org/licenses/MIT).
评论
    相关推荐
    • gitRepository
      gitRepository
    • git-repository
      git-repository
    • git003
      git003
    • HelloGit
      HelloGit
    • git003
      git003
    • gothrift:节俭来自 git.apache.orgthrift.gitlibgothrift
      Thrift Go 软件库 执照 根据一项或多项贡献者许可协议授权给 Apache 软件基金会 (ASF)。... $ go get git-wip-us.apache.org/repos/asf/thrift.git/lib/go/thrift 将检索并安装最新版本的软件包。 关于可选字段的
    • git003
      git003
    • git-scanner:用于代码分析的Git扫描器
      Git扫描仪 基于进行以下更改: 仅保留git逻辑。 发布到crates.io 添加git clone。 执照 版权所有2019 Kornelis Sietsma 版权所有:copyright:... 根据Apache许可证2.0版获得许可-有关详细信息,请参见LICENSE.txt
    • gitblit.zip
      gitblit 1.8 中文修改版 解决下载中文问题 和中文目录问题 安装方法:安装原版后替换相应的文件 ...如果是apache+tomcat方式,有乱码出现,将 JkOptions 屏蔽掉,并设置tomcat URL为UTF-8 url方式。
    • My apache setting
      My apache setting build up of svn and git