Skip to content

Commit f2395e0

Browse files
ygalnezriygalnezri
authored
Merge pull request #253 from thalesgroup-cert/test
Test
2 parents b26ddf2 + e6829f3 commit f2395e0

34 files changed

+1178
-58
lines changed

‎Watcher/README.md‎

Lines changed: 334 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -802,4 +802,337 @@ All data tables include **enhanced filtering**:
802802

803803
1. **Filter Builder**: Click filter icon to add conditions
804804
2. **Save Filter Sets**: Name and save custom filters
805-
3. **Quick Apply
805+
3. **Quick Apply**: Load saved filters with one click
806+
807+
## Archived Alerts
808+
Once you have processed an alert, you can archive it.
809+
810+
To archived **1** alert:
811+
812+
- Go to the alert that you want to archived.
813+
- Select the "**disable**" **button**.
814+
815+
To archived **several** alerts:
816+
817+
- Go to **/admin** page.
818+
- Click on **Alerts**.
819+
- **Check** alerts that you want to archived.
820+
- Click on **Action** dropdown.
821+
- Select "**Disable selected alerts**".
822+
823+
# Update Watcher
824+
825+
Verify that your local files `/.env`, `/docker-compose.yml` and `/Searx/` are **up-to-date**.
826+
827+
## Migrate from Docker Hub to GHCR (Required for all users)
828+
829+
If you are still using the old Docker Hub image (`felix83000/watcher`), you **must** update your `docker-compose.yml`:
830+
831+
**Old (DEPRECATED):**
832+
```yaml
833+
image: felix83000/watcher:latest
834+
```
835+
836+
**New (REQUIRED):**
837+
```yaml
838+
image: ghcr.io/thalesgroup-cert/watcher:latest
839+
```
840+
841+
## Update Process
842+
843+
To update Watcher image please follow the instructions below:
844+
845+
- Stop all containers: `docker compose down`
846+
- Remove the old docker images:
847+
```bash
848+
# If migrating from Docker Hub
849+
docker rmi felix83000/watcher
850+
851+
# If already using GHCR
852+
docker rmi ghcr.io/thalesgroup-cert/watcher
853+
854+
# Also remove Searx image
855+
docker rmi searx/searx
856+
```
857+
- Pull the newer docker images: `docker compose up`
858+
859+
This will update the Watcher project.
860+
861+
## Migrate & Populate the database (mandatory)
862+
863+
Updates the state of the database in accordance with all current models and migrations. Populate your database with hundred of banned words and RSS sources related to Cyber Security.
864+
865+
docker compose down
866+
docker compose run watcher bash
867+
python manage.py migrate
868+
python manage.py populate_db
869+
870+
## Managing Breaking Changes (optional)
871+
872+
If the release includes breaking changes, additional steps may be required. Here are the general steps to manage breaking changes:
873+
874+
1. **Review release notes or documentation**: Check the release notes or documentation for any information about breaking changes and specific instructions on how to handle them.
875+
876+
2. **Backup data**: Before proceeding with the update, it's advisable to backup any critical data to prevent loss in case of unexpected issues.
877+
878+
3. **Test in a staging environment**: If possible, test the update in a staging environment to identify any potential issues or conflicts with existing configurations or customizations.
879+
880+
4. **Update configurations**: Review and update any configurations or settings that may be affected by the breaking changes.
881+
882+
5. **Modify custom code**: If you have any custom code or scripts that rely on the previous version's functionality, you may need to modify them to work with the new version.
883+
884+
6. **Run additional migration scripts**: If provided, run any additional migration scripts or commands provided by the developers to handle specific breaking changes.
885+
886+
Always refer to the specific instructions provided with the update.
887+
888+
# Developers
889+
If you want to modify the project and Pull Request (PR) your work, you will need to setup your development environment.
890+
891+
## Open a Pull Request (PR) to contribute to this project
892+
- Fork the official Watcher repository
893+
- Install `Git`
894+
- Open a terminal: `git clone <your_forked_repository.git>`
895+
- Switch to the dev branch: `git checkout -b feature/<name_of_the_new_feature>`
896+
- Make your changes on the working files and then: `git add *`
897+
- Add a commit message and description: `git commit -m "Title" -m "Description"`
898+
- Publish the changes: `git push origin feature/<name_of_the_new_feature>`
899+
- Back to GitHub on your forked repository, click Under Contribute > Open Pull Request and then Confirm the operation
900+
- Done! Your work will be reviewed by the team!
901+
902+
## Setup Watcher environment
903+
Use a Linux server, we recommend the use of a Virtual Machine (Ubuntu 20.04 and Ubuntu 21.10 LTS in our case).
904+
905+
Then, follow the steps below:
906+
907+
- **Update and upgrade your machine:** `sudo apt update && sudo apt upgrade -y`
908+
- **Install Python and Node.js:** `sudo apt install python3 python3-pip -y` **&** `sudo apt install nodejs -y`
909+
- **Create and activate a Python virtual environment:** `python3 -m venv .venv |source .venv/bin/activate`
910+
- **Pull Watcher code:** `git clone <your_forked_repository.git>`
911+
- **Move to the following directory:** `cd Watcher/Watcher`
912+
- **Install** `python-ldap` **dependencies:** `sudo apt install -y libsasl2-dev python-dev-is-python3 libldap2-dev libssl-dev`
913+
- **Install** `mysqlclient` **dependency:** `sudo apt install default-libmysqlclient-dev`
914+
- **Install Rust (for tokenizers, etc)** `curl https://sh.rustup.rs -sSf | sh -s -- -y |source $HOME/.cargo/env`
915+
- **Install Python dependencies:** `pip3 install -r requirements.txt`
916+
- **Install Torch and Torchvision dependencies:** `pip install --extra-index-url https://download.pytorch.org/whl/cpu torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0`
917+
- **Install NLTK/punkt dependency:** `python3 ./nltk_dependencies.py`
918+
- If you have a proxy, you can configure it in `nltk_dependencies.py` script.
919+
- **Install Node.js dependencies:**
920+
- `sudo apt install npm -y`
921+
- `npm install`
922+
- **Install MySQL:**
923+
- `sudo apt install mysql-server -y`
924+
- `sudo mysql_secure_installation`
925+
- Enter root password.
926+
- You may now enter `Y` and `ENTER`. Accept all fields. This will remove some anonymous users and the test database,
927+
disable remote root logins, and load these new rules so that MySQL immediately respects any changes made.
928+
929+
**Create & Configure Watcher database:**
930+
931+
sudo mysql
932+
CREATE USER 'watcher'@'localhost' IDENTIFIED BY 'Ee5kZm4fWWAmE9hs!';
933+
GRANT ALL PRIVILEGES ON *.* TO 'watcher'@'localhost' WITH GRANT OPTION;
934+
CREATE DATABASE db_watcher;
935+
use db_watcher;
936+
exit
937+
systemctl status mysql.service
938+
939+
- `cd Watcher/watcher`
940+
941+
In `settings.py` change `HOST` variable to `localhost`:
942+
943+
DATABASES = {
944+
'default': {
945+
'ENGINE': 'django.db.backends.mysql',
946+
'CONN_MAX_AGE': 3600,
947+
'NAME': 'db_watcher',
948+
'USER': 'watcher',
949+
'PASSWORD': 'Ee5kZm4fWWAmE9hs!',
950+
'HOST': 'localhost',
951+
'PORT': '3306',
952+
'OPTIONS': {
953+
'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
954+
},
955+
}
956+
}
957+
958+
- <span style="color:red">**[IMPORTANT]** When **commit** put `HOST` variable back to `db_watcher`</span>
959+
- `cd ..`
960+
- **[Migrate](#migrate) the database:** `python3 manage.py migrate`
961+
- **Run Watcher:** `python3 manage.py runserver`
962+
963+
## Deploy a simple SMTP server to test the email notifications
964+
If you are working on a test environment and willing to have email alerts, here is a simple way to configure the SMTP settings to make it work.
965+
- Grab the docker-compose file: [here](https://github.com/rnwood/smtp4dev/blob/master/docker-compose.yml).
966+
- Run the command: `docker compose up`
967+
- The mails will be available here by default: `localhost:5000`
968+
- Modify the mail settings in the environment variables: `vi /.env`
969+
- `SMTP_SERVER=localhost`
970+
- `[email protected]`
971+
- Launch Watcher: `python Watcher/Watcher/manage.py runserver`
972+
973+
## Unit Testing
974+
975+
Watcher includes comprehensive unit tests to ensure code quality and reliability. **When contributing new features, you must include corresponding unit tests.**
976+
977+
### Test Coverage & Technologies
978+
979+
The test suite covers all 5 main modules of Watcher:
980+
981+
- **Back-End Tests**: Python's standard unittest module
982+
- `common/tests.py`
983+
- `watcher/tests.py`
984+
- Module-specific test files for each component
985+
986+
- **Front-End Tests**: Cypress framework for JavaScript testing
987+
- Cypress tests for all 5 modules
988+
- End-to-end testing scenarios
989+
990+
### Automated CI/CD Testing
991+
992+
All tests are **automatically executed** in our CI/CD pipeline using **GitHub Actions**:
993+
994+
- **Triggered on**: Push, Pull Requests, and manual workflow dispatch
995+
- **Execution**: Both back-end and front-end tests run automatically
996+
- **Coverage**: Full test suite validation before code integration
997+
998+
The CI/CD workflow ensures that:
999+
- No broken code reaches the main branch
1000+
- All new features are properly tested
1001+
1002+
You can view the workflow file at `.github/workflows/docker-image-latest.yml` and monitor test results in the **Actions** tab of the GitHub repository.
1003+
1004+
1005+
### The commands
1006+
1007+
**IMPORTANT**: All test commands must be executed from the `Watcher/Watcher` directory:
1008+
1009+
```bash
1010+
cd Watcher/Watcher
1011+
```
1012+
1013+
#### Back-End Tests
1014+
To run all Django unit tests:
1015+
1016+
```bash
1017+
python manage.py test
1018+
```
1019+
1020+
To run tests for a specific module:
1021+
1022+
```bash
1023+
python manage.py test <module_name>
1024+
```
1025+
1026+
To run with verbose output:
1027+
1028+
```bash
1029+
python manage.py test --verbosity=2
1030+
```
1031+
1032+
#### Front-End Tests
1033+
1034+
Before running front-end tests, you need to create a test superuser:
1035+
1036+
```bash
1037+
python manage.py shell -c "
1038+
from django.contrib.auth.models import User
1039+
User.objects.create_superuser('Watcher', '[email protected]', 'Watcher', first_name='Unit-Test Cypress', last_name='Watcher')
1040+
"
1041+
```
1042+
1043+
**Alternative**: If you already have a superuser account, you can use it for testing by updating the test credentials in `cypress.config.js`:
1044+
1045+
```javascript
1046+
env: {
1047+
testCredentials: {
1048+
username: 'your_superuser_name',
1049+
password: 'your_superuser_password',
1050+
email: 'your_superuser_email',
1051+
firstName: 'your_first_name'
1052+
}
1053+
}
1054+
```
1055+
1056+
To run with an interactive Cypress Test Runner:
1057+
1058+
```bash
1059+
npm run cypress:open
1060+
```
1061+
1062+
To run in headless mode (CI/CD):
1063+
1064+
```bash
1065+
npm run test:e2e
1066+
```
1067+
1068+
### Development Guidelines
1069+
1070+
Here is the file structure of the test files in Watcher:
1071+
```
1072+
Watcher/
1073+
├── common/
1074+
│ └── tests.py
1075+
├── watcher/
1076+
│ └── tests.py
1077+
├── [module_name]/
1078+
│ └── tests.py
1079+
└── cypress/
1080+
├── e2e/
1081+
└── support/
1082+
```
1083+
1084+
- **For new modules**, create a `tests.py` file following Django's testing conventions.
1085+
- **For frontend features**, add Cypress tests in the appropriate `cypress/e2e/` directory.
1086+
1087+
**When developing new features**, ensure your tests cover:
1088+
- Happy path scenarios
1089+
- Edge cases
1090+
- Error handling
1091+
- Input validation
1092+
1093+
<span style="color:red">**[IMPORTANT]** All Pull Requests must include tests for new functionality. PRs without adequate test coverage may be rejected.</span>
1094+
1095+
## Modify the frontend
1096+
If you need to modify the frontend `/Watcher/Watcher/frontend`:
1097+
1098+
From `/Watcher/Watcher/`, run the command below:
1099+
1100+
npm run dev
1101+
1102+
Let this command run in background.
1103+
Now, when modifying some frontend ReactJs files it will automatically build them into one file (`/Watcher/Watcher/frontend/static/frontend/main.js`).
1104+
1105+
<span style="color:red">**[IMPORTANT]** When **commit** you have to run **1 time** the command below:</span>
1106+
1107+
npm run build
1108+
1109+
## Migrations: Change the database schema
1110+
Migrations are Django’s way of propagating changes you make to your models
1111+
(adding a field, deleting a model, etc.) into your database schema. They’re designed to be mostly automatic,
1112+
but you’ll need to know when to make migrations, when to run them, and the common problems you might run into.
1113+
1114+
### The commands
1115+
There are several commands which you will use to interact with migrations and Django’s handling of database schema:
1116+
- `migrate`, which is responsible for applying and unapplying migrations.
1117+
- `makemigrations`, which is responsible for creating new migrations based on the changes you have made to your models.
1118+
- `sqlmigrate`, which displays the SQL statements for a migration.
1119+
- `showmigrations`, which lists a project’s migrations and their status.
1120+
1121+
### Change a model (adding a field, deleting a model, etc.)
1122+
When you are **making a change to a model**, for instance, adding a new field to: **/Watcher/Watcher/data_leak/models.py**
1123+
Then, you need to create a new migration based on the changes you have made:
1124+
- Go to **/Watcher/Watcher/** and run this command: `python3 manage.py makemigrations`
1125+
1126+
<span style="color:red"> **[IMPORTANT]** Run the `makemigrations` command **only once**</span>, when you have made **all the changes**.
1127+
Otherwise, it will create **several unnecessary migration files**.
1128+
1129+
## Build the documentation
1130+
Modify some function comments or the `/Watcher/README.md` file.
1131+
1132+
Go to `/Watcher/docs` and run:
1133+
1134+
./build_the_docs.sh
1135+
1136+
When commit please add the all `/Watcher/docs` folder and the `README.md` file:
1137+
1138+
git add ../docs ../README.md

0 commit comments

Comments
 (0)