Tibco/Capiot Model Evaluation
1. Required Input Files
You must have at least three CSV files for the purpose of this evaluation.
- Gold-standard file with true labels.
- Output of model A.
- Output of model B.
Files (2) and (3) must be of the following format:
| id | category | p1 | ... | p21 |
|---|---|---|---|---|
| 34943 | 1 | 0.92 | ... | 0.11 |
| 45234 | 3 | 0.32 | ... | 0.23 |
| 63144 | 2 | 0.08 | ... | 0.95 |
The gold-standard file (1) will not have the p columns. These include the correct
unique id/category classifications.
2. Execution
The code to execute the comparison is in comparison.py. It was developed in Python 3.6.x, so it is recommended that you have the same version or above. See the requirements.txt file to find required packages.
To run:
python comparison.py <gs_file.csv> <file_1.csv> ... <file_n.csv>
Use the verbose flag -v to see a summary of the macro-level statistics per category.
For more information about arguments, run:
python comparison.py -h