Skip to content

Added anomaly detection script and requirements file - #140

Open
Swastik (swastik-21) wants to merge 1 commit into
microsoft:masterfrom
swastik-21:feature/anomaly-detection
Open

Added anomaly detection script and requirements file#140
Swastik (swastik-21) wants to merge 1 commit into
microsoft:masterfrom
swastik-21:feature/anomaly-detection

Conversation

@swastik-21

Copy link
Copy Markdown

Hi,
I’ve added a simple anomaly detection script under ai_modules/anomaly_detection.py. Right now it uses a z-score approach to flag unusual values in water usage data. The threshold can be adjusted so it works with different datasets.
I also created a requirements.txt so dependencies can be installed more easily.
Tested locally with sample data, and it correctly flagged extreme outliers. This is an initial version, so open to feedback or suggestions on improvements.
Thanks

@swastik-21

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few suggestions for improvements. Overall looks good!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The anomaly detection function is clear and well-documented. A few suggestions:

  1. Consider adding a check for std == 0 to avoid division by zero errors.
  2. In the example usage, it might help to explain why z_thresh=2.0 is chosen (e.g., more sensitive to outliers).
  3. Optionally, you could mention that this method assumes a roughly normal distribution for data.

Comment thread requirements.txt

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requirements.txt is clear and version-pinned, which is great for reproducibility. A few suggestions:

  1. Double-check if numpy==2.3.2 and pandas==2.3.2 are compatible; sometimes older versions of one library may conflict with the other.
  2. Consider whether all dependencies (like six or tzdata) are actually used in the project — removing unused packages can simplify the environment.
  3. Optionally, you could mention Python version compatibility, e.g., if this works for Python 3.11 or higher.

@ManyaS-Git Manya Sharma (ManyaS-Git) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, small and readable sample — a good first contribution. The Z-score approach is a clean choice for univariate water-usage data. A few issues to consider before this is merge-ready:

1. Division by zero on constant input (detect_anomalies).
std = data.std() is 0.0 whenever all values are identical, and (data - mean) / std then produces NaN (plus a NumPy RuntimeWarning). Since abs(NaN) > z_thresh is False, the function silently returns an empty result for a legitimately flat series. The same happens for single-element input (std is NaN with the default ddof=1). Recommend guarding, e.g.:

if std == 0:
    return data[data != mean]  # or pd.Series([], dtype=data.dtype)

or delegating to scipy.stats.zscore, which returns 0.0 for constant data.

2. ddof=1 vs population Z-score.
Series.std() defaults to sample standard deviation (ddof=1), while the classic Z-score for a full dataset uses population std (ddof=0). With small samples this inflates the Z-scores and can push legitimately normal points past the threshold. Either is defensible, but it should be a deliberate choice — consider passing ddof=0 or noting the convention in the docstring.

3. NaN in the input silently disables detection.
If the series contains any NaN, mean/std become NaN and all comparisons evaluate False, so the function reports no anomalies without warning. Suggest data = data.dropna() at the top (documented) so missing readings don't silently swallow the check.

4. Root-level requirements.txt with exact pins of transitive deps.
The repo root currently has no requirements.txt; this adds one that pins numpy, python-dateutil, pytz, six, and tzdata to exact versions — all transitive dependencies of pandas that the sample never imports directly. Exact == pins of transitive packages are fragile and could conflict with other samples/modules later added to the repo. Prefer pinning only what's actually used with a floor bound, e.g. pandas>=2.0, and drop the transitive entries (or generate a dedicated requirements.txt inside the sample folder instead of at the repo root).

5. New top-level ai_modules/ folder.
The repo doesn't have an ai_modules/ directory today; introducing a generic top-level folder with a broad name doesn't match the existing layout (ai100-samples, ai200-architectures, ai300-practices, utilities, ...). Consider placing this under an existing area or a self-describing directory (e.g. ai300-practices/anomaly-detection-zscore/) so the sample is discoverable alongside the other examples.

6. Minor: the function returns the anomalous values rather than their indices/positions; the docstring says "points", which is ambiguous. Returning the boolean mask or the indices is often more useful for plotting/flagging, so clarify the contract in the docstring. The __main__ demo works correctly with z_thresh=2.0 (the 500 spike is flagged).

Overall a solid, focused PR — addressing items 1, 3, and 4 would be the most impactful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants