Repository navigation
Fix open issues: faster lookups, --color, multi-DB queries, update skipping, .deb packages - #215
Merged
Merged
Conversation
Every run read the whole database into memory with io.ReadAll, which grows its buffer by doubling: loading the 27 MB qqwry.dat allocated and copied ~55 MB and triggered 4 GC cycles, although a lookup only touches a few pages. On slow routers this is where the reported 7-10 seconds per query went (#195). qqwry, zxipv6wry and ip2region now memory-map their files on Linux, macOS and FreeBSD (falling back to os.ReadFile elsewhere or on error). `nali 223.5.5.5` drops from ~160 ms to ~10 ms on x86 with no GC cycles; results are identical for 3000 random IPs on qqwry and ip2region. Mappings are unaffected by `nali update`, which replaces files by rename. Fixes #195 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
--color auto|always|never (default auto). Colors were only emitted when stdout is a terminal, so `cmd1 | nali | cmd2` lost them (#209); `never` also helps terminals where the colors are unreadable (#205). In auto mode NO_COLOR is honored as before. On Windows consoles using the GBK code page (936/54936), input lines that are not valid UTF-8 are now decoded as GBK automatically. Output of tools such as tracert or nslookup was shown garbled unless --gbk was passed (#208, #185). Valid UTF-8 input is left untouched and --gbk still forces GBK decoding everywhere. Fixes #209 Fixes #208 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
`nali update` downloaded every database again even when nothing changed (#190). It now sends If-Modified-Since with the local file's modification time and stops on 304 Not Modified; for servers that ignore the header, a download identical to the local file is not written again. Either way it reports "<name> 数据库已是最新版本". Fixes #190 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
`make deb` builds Debian packages for amd64, i386, armhf and arm64 with dpkg-deb (preinstalled on GitHub's Ubuntu runners, no new tooling). The release workflow builds them before compressing the binaries, so they are attached to the GitHub release and uploaded with the other assets. Fixes #154 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
A selection may now list several databases separated by commas, e.g.
`NALI_DB_IP4=qqwry,ipinfo` or `selected.ipv4: qqwry,ipinfo`. Each
database is queried and its result shown in its own brackets:
8.8.8.8 [美国 谷歌公司DNS服务器] [United States AS15169 Google LLC]
JSON keeps the first database's result in source/text/info, and adds a
"results" array with every database's answer only when more than one
database is selected, so existing consumers are unaffected. A plain
`nali update` also refreshes optional databases that appear anywhere in
such a list.
Fixes #127
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
- Debian/Ubuntu install from the release .deb packages - --color and the updated --gbk description in the help output - querying several databases at once and the JSON "results" array - up-to-date databases are not downloaded again - FAQ: colors in pipelines, garbled text on Windows, `jq --unbuffered` when nali follows jq (#204), filtering CDN matches with --json (#160) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
Owner
Author
|
No code fix applies. I couldn't re-run the job from here ( Generated by Claude Code |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL: `db == nil` can never be true because db holds a typed pointer wrapped in the dbif.DB interface. Every constructor returns a non-nil error whenever it returns a nil database, so checking err is sufficient. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Conditional updates can retain stale data, and multi-database output and package versioning have unresolved correctness issues.
Review effort: Balanced
Findings: 2
Open (7)
What changed in this PR
This PR addresses database lookup speed, output options, update behavior, and Debian packaging in nali.
Changes:
- Map large database files and support multiple database results.
- Add
--color, Windows input decoding, and conditional database updates. - Build
.debpackages and document the new behavior.
| File | Description |
|---|---|
| README.md | Documents options, updates, packages, and FAQs. |
| README_en.md | Adds the corresponding English guidance. |
| pkg/zxipv6wry/zxipv6wry.go | Uses mapped database loading. |
| pkg/qqwry/qqwry.go | Uses mapped database loading. |
| pkg/ip2region/ip2region.go | Uses mapped database loading. |
| pkg/entity/parse.go | Collects multiple lookup results. |
| pkg/entity/entity.go | Renders and serializes multiple results. |
| pkg/entity/entity_test.go | Tests multi-result output. |
| pkg/common/mmap_unix.go | Adds Unix memory-mapped loading. |
| pkg/common/mmap_other.go | Adds a read-file fallback. |
| pkg/common/httpclient.go | Adds conditional HTTP requests. |
| pkg/common/encoding.go | Adds input decoding logic. |
| pkg/common/console_windows.go | Detects Windows console code pages. |
| pkg/common/console_other.go | Provides non-Windows detection behavior. |
| pkg/common/common_test.go | Tests loading, decoding, and HTTP responses. |
| pkg/cdn/cdn.go | Simplifies CDN file reading. |
| Makefile | Adds the Debian package target. |
| internal/db/update.go | Adds update checks and multi-selection handling. |
| internal/db/update_test.go | Tests unchanged-download paths. |
| internal/db/db.go | Queries multiple selected databases. |
| internal/db/db_test.go | Tests multi-database lookup and selection. |
| internal/db/cache.go | Caches database lists. |
| go.mod | Makes the Windows system package a direct dependency. |
| cmd/root.go | Adds color and input-decoding options. |
| .github/workflows/go-release.yml | Builds packages for releases. |
| .github/scripts/build_deb.sh | Assembles Debian packages. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…30 and deb versions - update: drop If-Modified-Since. The local mtime is when nali saved the file, not the server's Last-Modified, so a server whose new content carries an older timestamp could answer 304 and keep a stale database. Databases are now always downloaded and an identical download is not written again. This also removes the path where an unsolicited 304 was taken as success, and a 304 is now an error like any non-200 status. - update: apply the same unchanged-content check to ZX IPv6, which uses its own downloader. - multi-DB: Find returns one entry per selected database (failed lookups included). Top-level JSON fields come from the first database with a non-empty answer; "results" is present whenever several databases are selected and lists failed lookups with empty text and null info. - Windows input detection decodes GB18030 (a superset of GBK), so four-byte characters on code page 54936 are decoded correctly. - deb versions: prereleases map to ~ (v0.8.1-rc.1 -> 0.8.1~rc.1, sorting before 0.8.1), git-describe suffixes keep +N.gHASH, and untagged builds use 0.0.0+unknown instead of an invalid version containing a space. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the open issues that can be solved in code, without new dependencies or external data. There is one commit per issue, plus two commits addressing review findings.
os.ReadFileelsewhere) instead of read withio.ReadAll.io.ReadAllcopied ~55 MB and ran 4 GC cycles to load the 27 MBqqwry.daton every run.--color auto|always|never(defaultauto, which still honorsNO_COLOR).--gbkstill forces GBK decoding.数据库已是最新版本. This covers every database, including ZX IPv6. Databases are still downloaded each time: conditional requests based on the local mtime were dropped after review, because the mtime is not a server validator.NALI_DB_IP4=qqwry,ipinfo/selected.ipv4: qqwry,ipinfo. Text output has one bracket per database. In JSON,source/text/infocome from the first database with an answer. Whenever several databases are selected, aresultsarray lists every one in order, with emptytextandnullinfofor lookups that failed. Single-DB output is unchanged..debpackagesmake debbuilds amd64/i386/armhf/arm64 packages withdpkg-deb, which is preinstalled on GitHub's Ubuntu runners. The release workflow attaches them to releases. Versions sort correctly:v0.8.1-rc.1→0.8.1~rc.1<0.8.1<0.8.1+14.gabc123, and untagged builds use0.0.0+unknown.tail -f | jq . | naliholds back linesjq --unbuffered."source": "cdn"in--jsonoutput (jq 'select(.source=="cdn")'). IP results have no reliable CDN flag, so this is not closed.Already fixed on master before this PR: #212 (ip2region v4/v6, via #210) and #158 (IPinfo, via #213).
Closes #212
Closes #158
Verification
go test -race ./...passes on Go 1.27.1, andgo test ./...passes on Go 1.26.0.go vetis clean for windows/darwin, actionlint passes,go mod tidy -diffis clean, and all 19 release targets build.nali 223.5.5.5drops from ~160 ms to ~10 ms on x86 with zero GC cycles. Output is byte-identical for 3,002 random IPs on qqwry and on ip2region.nali updatereports the unchanged databases as up to date.NALI_DB_IP4=cdn,ipinfo,qqwryshows the answering databases, and the failedcdnlookup appears inresultswithnullinfo. With a single database, text and JSON output are byte-identical to master..debpackage support for Debian/Ubuntu #154: installed the amd64 package withdpkg -i, ran/usr/bin/nali, and removed it withdpkg -r. Version ordering was checked withdpkg --compare-versions.Refs, notFixes, for those issues.claude-reviewfails on this PR because the action's Claude run exits immediately with zero model usage. The cause is theCLAUDE_CODE_OAUTH_TOKENsecret or its quota, not the diff; see the comment below.1ac7e40has aFixes #208trailer. Please remove it from the squash message so #208 is not auto-closed before someone confirms the fix on Windows.Fixes #209in the same commit is intended.Not addressed (need external data, tooling or a maintainer decision)
qqwry.datsource that master uses by default. Commenters confirmed it works, so it can probably be closed.🤖 Generated with Claude Code
https://claude.ai/code/session_0143Y5sXSR6czPJc5MuC37Xw