Spamhaus ipv6 and asn support #2761

pranjalg1331 · 2025-02-20T10:15:19Z

(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example: <feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)

Description

closes #2408
Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
[ x] New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review by using GitHub's reviewing system detailed here.

pranjalg1331 · 2025-02-22T17:26:11Z

pranjalg1331 · 2025-02-22T17:27:07Z

@mlodic, I have added the support for ASN and IPv6. Please review.

fgibertoni · 2025-02-24T07:51:07Z

@pranjalg1331 please use the appropriate GitHub system to ask for a review and avoid directly tagging if not strictly necessary.
Also, the PR is not ready as there are still some errors in the CI that should be addressed.

pranjalg1331 · 2025-02-24T08:14:40Z

I am unsure whether I am authorized to request reviews from the maintainer using Github ?

pranjalg1331 · 2025-02-24T08:20:15Z

Invalid observable: [email protected]
The test failure is because an invalid parameter is being passed for analysis. Spamhaus does not support the analysis of URLs. Could you please suggest how can I rectify this error?

mlodic · 2025-02-24T08:27:31Z

About reviews: Github introduced a new "moderation settings" that would prevent non-organization members to ask for reviews. I have just removed that limit. Please confirm that now you can.

About URL support: like we did in many other analyzers, just extract the domain from the URL and analyze it

mlodic · 2025-02-24T08:30:29Z

You should also be able to tag the PR as draft instead of writing it in the name of the PR

mlodic · 2025-02-24T08:46:31Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

    url = "https://www.spamhaus.org/drop/drop_v4.json"
+    ipv6_url = "https://www.spamhaus.org/drop/drop_v6.json"
+    asn_url = "https://www.spamhaus.org/drop/asndrop.json"


use a url parameter which is the base to reuse the code + to use it for health checks

https://www.spamhaus.org/drop

mlodic · 2025-02-24T08:48:55Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

+        if self.observable_name.isdigit():  # If it's numeric, treat it as an ASN
+            data_type = "asn"
+            asn = int(self.observable_name)  # Convert to integer
+            logger.info(f"The given observable is an ASN: {asn}")
+        else:
+            try:
+                ip = ipaddress.ip_address(self.observable_name)
+                data_type = "ipv4" if ip.version == 4 else "ipv6"
+                logger.info(f"The given observable is an {data_type} address.")
+            except ValueError:
+                raise ValueError(f"Invalid observable: {self.observable_name}")


you can use this to check the observable type and then, do the related checks

from api_app.choices import Classification

mlodic · 2025-02-24T10:02:17Z

I would like to accelerate this and finish this as soon as possible so @cristinaascari can finish her refactor here https://github.com/intelowlproject/IntelOwl/pull/2767/files. Please @pranjalg1331, once you have some time, prioritize this. Thanks

pranjalg1331 · 2025-02-24T13:29:47Z

About reviews: I can re-ask for your review in this pr, but still cannot ask any other maintainer for a review.

pranjalg1331 · 2025-02-24T13:32:36Z

About URL support: like we did in many other analyzers, just extract the domain from the URL and analyze it

Spamhaus does not work with domains either. Since it's now in the generic category too, [email protected] is being provided as a sample observable. Since it's not a valid ASN value, the test case is failing.

an5 = Analyzable.objects.create( name="[email protected]", classification=Classification.GENERIC, )

mlodic · 2025-02-24T16:27:41Z

Spamhaus does not work with domains either. Since it's now in the generic category too, [email protected] is being provided as a sample observable. Since it's not a valid ASN value, the test case is failing.

Ok so you can change those tests and make custom ones for this use case

pranjalg1331 · 2025-02-24T19:11:01Z

I have added a new value for Spamhaus test.

fgibertoni · 2025-03-03T07:40:54Z

api_app/analyzers_manager/migrations/0152_alter_spamhaus_drop_supported_observable.py

+class Migration(migrations.Migration):
+    atomic = False
+    dependencies = [
+        ("analyzers_manager", "0151_analyzer_config_ipquery"),


Please update your migration number and its dependencies to reflect latest changes in `develop branch

api_app/analyzers_manager/migrations/0152_alter_spamhaus_drop_supported_observable.py

fgibertoni · 2025-03-03T07:59:17Z

api_app/analyzers_manager/migrations/0152_alter_spamhaus_drop_supported_observable.py

+        if "generic" in observable_supported:
+            observable_supported.remove("generic")
+            obj.observable_supported = observable_supported
+            obj.save()


Suggested change

obj.save()

obj.full_clean()

obj.save()

fgibertoni · 2025-03-03T08:02:32Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

+            db_name = "drop_v4.json"
+        elif data_type == "ipv6":
+            db_name = "drop_v6.json"
+        else:


Suggested change

else:

elif data_type == "asn":

Then you can add an else clause that throws an error if the data_type is not supported, logging what was passed.

fgibertoni · 2025-03-03T08:04:48Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

+        if self.observable_classification == Classification.IP:
+            ip = ipaddress.ip_address(self.observable_name)
+            data_type = "ipv4" if ip.version == 4 else "ipv6"
+            logger.info(f"The given observable is an {data_type} address.")


Also log the observable for tracing pls

fgibertoni · 2025-03-03T08:05:47Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

-    def location(cls) -> str:
-        db_name = "drop_v4.json"
+    def location(cls, data_type: str) -> str:
+        print("location", data_type)


Why a print ?

fgibertoni · 2025-03-03T08:06:10Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

+        ):
+            data_type = "asn"
+            asn = int(self.observable_name)  # Convert to integer
+            logger.info(f"The given observable is an ASN: {asn}")


This log is still missing. This should be as on line 41-42

fgibertoni · 2025-03-03T08:09:46Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

+                    matches.append(db[i])
+                elif network.network_address > ip:
+                    break
+        else:


Suggested change

else:

elif data_type == "asn":

and same logging as in location()

Signed-off-by: pranjalg1331 <[email protected]>

…supported_observable.py Co-authored-by: Federico Gibertoni <[email protected]>

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 · 2025-03-04T07:00:05Z

I have made the requested changes.

fgibertoni · 2025-03-04T09:06:53Z

api_app/analyzers_manager/observable_analyzers/spamhaus_drop.py

+            db_url = cls.asn_url
+        else:
+            raise AnalyzerRunException(
+                "Invalid type provided for updating the database"


Also log the value of the wrong data_type pls

fgibertoni · 2025-03-04T09:07:37Z

After these small changes it's good to go by my side! Thank you

Signed-off-by: pranjalg1331 <[email protected]>

fgibertoni

Thank you!

pranjalg1331 · 2025-03-14T13:11:20Z

Can we merge this pr?

pranjalg1331 force-pushed the spamhaus_ipv6 branch 2 times, most recently from 70eb6f1 to b9cd155 Compare February 22, 2025 17:03

mlodic requested changes Feb 24, 2025

View reviewed changes

mlodic mentioned this pull request Feb 24, 2025

refactor of analyzers for local queries #2767

Closed

25 tasks

pranjalg1331 marked this pull request as draft February 24, 2025 10:53

pranjalg1331 requested a review from mlodic February 24, 2025 13:33

pranjalg1331 changed the title ~~Spamhaus ipv6 and asn support (Draft)~~ Spamhaus ipv6 and asn support Feb 26, 2025

pranjalg1331 marked this pull request as ready for review February 26, 2025 10:18

fgibertoni requested changes Mar 3, 2025

View reviewed changes

pranjalg1331 added 9 commits March 4, 2025 10:51

ipv6 working

02bcbf0

Signed-off-by: pranjalg1331 <[email protected]>

asn working

478e9ff

Signed-off-by: pranjalg1331 <[email protected]>

updates

8c5a3ae

Signed-off-by: pranjalg1331 <[email protected]>

updates2

bfee8b3

Signed-off-by: pranjalg1331 <[email protected]>

migration file

881d078

Signed-off-by: pranjalg1331 <[email protected]>

correct migration change

0f09fd7

Signed-off-by: pranjalg1331 <[email protected]>

update

c89a246

Signed-off-by: pranjalg1331 <[email protected]>

healthcheck_url

741ab35

Signed-off-by: pranjalg1331 <[email protected]>

using Classification class

b3ad0b7

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 and others added 4 commits March 4, 2025 10:51

test case

308d21f

Signed-off-by: pranjalg1331 <[email protected]>

Update api_app/analyzers_manager/migrations/0152_alter_spamhaus_drop_…

851fc51

…supported_observable.py Co-authored-by: Federico Gibertoni <[email protected]>

updates

e0ecfc0

Signed-off-by: pranjalg1331 <[email protected]>

migration number

3f22e1b

Signed-off-by: pranjalg1331 <[email protected]>

pranjalg1331 force-pushed the spamhaus_ipv6 branch from 3c3d2fb to 3f22e1b Compare March 4, 2025 05:23

pranjalg1331 requested a review from fgibertoni March 4, 2025 07:00

fgibertoni reviewed Mar 4, 2025

View reviewed changes

logs

dbe5b8c

Signed-off-by: pranjalg1331 <[email protected]>

fgibertoni approved these changes Mar 4, 2025

View reviewed changes

fgibertoni merged commit 24c79d6 into intelowlproject:develop Mar 14, 2025
10 checks passed

Uh oh!

Spamhaus ipv6 and asn support #2761

Spamhaus ipv6 and asn support #2761

Uh oh!

Conversation

pranjalg1331 commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Important Rules

Uh oh!

pranjalg1331 commented Feb 22, 2025

Uh oh!

pranjalg1331 commented Feb 22, 2025

Uh oh!

fgibertoni commented Feb 24, 2025

Uh oh!

pranjalg1331 commented Feb 24, 2025

Uh oh!

pranjalg1331 commented Feb 24, 2025

Uh oh!

mlodic commented Feb 24, 2025

Uh oh!

mlodic commented Feb 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlodic commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pranjalg1331 commented Feb 24, 2025

Uh oh!

pranjalg1331 commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlodic commented Feb 24, 2025

Uh oh!

pranjalg1331 commented Feb 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pranjalg1331 commented Mar 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fgibertoni commented Mar 4, 2025

Uh oh!

fgibertoni left a comment

Choose a reason for hiding this comment

Uh oh!

pranjalg1331 commented Mar 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

pranjalg1331 commented Feb 20, 2025 •

edited

Loading

mlodic commented Feb 24, 2025 •

edited

Loading

pranjalg1331 commented Feb 24, 2025 •

edited

Loading