Beyond coding. We forge.
Find a file
wxiaoguang 6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
.gitea Issue template form (#16349) 2021-09-15 20:33:13 +03:00
.github Check latest version on CI (#24556) 2023-05-06 15:43:56 +08:00
assets Add RPM registry (#23380) 2023-05-05 20:33:37 +00:00
build Replace placeholders in licenses (#24354) 2023-05-05 13:46:17 +00:00
cmd Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
contrib Updated upgrade script that is informing user that Gitea service has to be running in order to upgrade it (#24260) 2023-04-25 13:19:58 -04:00
custom/conf Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
docker Unify nightly naming across binaries and docker images (#24116) 2023-04-24 23:43:19 +08:00
docs Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
models Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
modules Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
options [skip ci] Updated licenses and gitignores 2023-05-08 00:23:39 +00:00
public Add RPM registry (#23380) 2023-05-05 20:33:37 +00:00
routers Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
services Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
snap Remove unnecessary whitespace in snapcraft.yaml (#22090) 2022-12-10 08:31:16 -06:00
templates Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
tests Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
web_src Add goto issue id function (#24479) 2023-05-07 23:44:16 +08:00
.air.toml Set stop_on_error (#24453) 2023-04-30 18:31:16 -04:00
.changelog.yml Changelog for v1.15.0-rc1 (#16422) 2021-07-15 11:47:57 -04:00
.dockerignore Add .dockerignore (#21753) 2022-11-10 04:04:09 +01:00
.drone.yml Upgrade to Node 20 on CI, enable actions cancellation (#24524) 2023-05-04 22:21:48 +00:00
.editorconfig Add markdownlint (#20512) 2022-07-28 09:22:47 +08:00
.eslintrc.yaml Add eslint-plugin-regexp (#24361) 2023-04-26 22:08:16 -04:00
.gitattributes Mark /templates/swagger/v1_json.tmpl as generated file (#24306) 2023-04-24 13:18:24 -04:00
.gitignore Remove git sample files and ignore them (#24271) 2023-04-22 20:29:29 +08:00
.gitpod.yml Remove docs from automatically building in Gitpod (#23978) 2023-04-07 15:55:25 -04:00
.golangci.yml Enable forbidigo linter (#24278) 2023-04-24 05:50:58 -04:00
.ignore Add some .ignore entries (#18296) 2022-01-16 17:26:15 +00:00
.markdownlint.yaml Add markdownlint (#20512) 2022-07-28 09:22:47 +08:00
.npmrc Upgrade to npm lockfile v3 and explicitely set it (#23561) 2023-03-18 19:38:10 +01:00
.spectral.yaml Add spectral linter for Swagger (#20321) 2022-07-11 18:07:16 -05:00
.stylelintrc.yaml Enhance stylelint rule config, remove dead CSS (#24472) 2023-05-02 23:15:52 -04:00
BSDmakefile update BSDmakefile to latest version from upstream (#24063) 2023-04-11 23:42:22 -04:00
build.go User/Org Feed render description as per web (#23887) 2023-04-04 04:39:47 +01:00
CHANGELOG.md Changelog for 1.19.3 (#24495) (#24506) 2023-05-03 22:05:47 -04:00
CODE_OF_CONDUCT.md Add Gitea Community Code of Conduct (#23188) 2023-03-09 10:49:34 +08:00
CONTRIBUTING.md Remove .lgtm and reference to old bot (#24464) 2023-05-01 08:49:30 -04:00
DCO Remove address from DCO (#22595) 2023-01-24 18:52:38 +00:00
Dockerfile Hardcode path to docker images (#23955) 2023-04-07 02:41:49 -04:00
Dockerfile.rootless Hardcode path to docker images (#23955) 2023-04-07 02:41:49 -04:00
go.mod Add RPM registry (#23380) 2023-05-05 20:33:37 +00:00
go.sum Add RPM registry (#23380) 2023-05-05 20:33:37 +00:00
LICENSE Fix typo 2016-11-08 08:42:05 +01:00
main.go add CLI command to register runner tokens (#23762) 2023-04-17 13:07:13 -04:00
MAINTAINERS Apply to become maintainer (#24254) 2023-04-21 21:23:16 +08:00
Makefile Makefile: Use portable !, not GNUish -not, with find(1). (#24565) 2023-05-07 12:41:33 +08:00
package-lock.json Remove font-awesome and fomantic icon module (#24471) 2023-05-01 13:25:54 -04:00
package.json Remove font-awesome and fomantic icon module (#24471) 2023-05-01 13:25:54 -04:00
playwright.config.js Update JS dependencies and eslint config (#21388) 2022-10-10 20:02:20 +08:00
README.md Fix broken README link (#24546) 2023-05-06 16:16:06 +08:00
README_ZH.md link update in README files (#22582) 2023-01-23 15:57:57 -05:00
SECURITY.md Add markdownlint (#20512) 2022-07-28 09:22:47 +08:00
vitest.config.js Update JS dependencies (#24218) 2023-04-19 22:36:41 -04:00
webpack.config.js Remove font-awesome and fomantic icon module (#24471) 2023-05-01 13:25:54 -04:00

Gitea

Gitea - Git with a cup of tea

Contribute with Gitpod

View this document in Chinese

Purpose

The goal of this project is to make the easiest, fastest, and most painless way of setting up a self-hosted Git service.

As Gitea is written in Go, it works across all the platforms and architectures that are supported by Go, including Linux, macOS, and Windows on x86, amd64, ARM and PowerPC architectures. You can try it out using the online demo. This project has been forked from Gogs since November of 2016, but a lot has changed.

Building

From the root of the source tree, run:

TAGS="bindata" make build

or if SQLite support is required:

TAGS="bindata sqlite sqlite_unlock_notify" make build

The build target is split into two sub-targets:

  • make backend which requires Go Stable, required version is defined in go.mod.
  • make frontend which requires Node.js LTS or greater and Internet connectivity to download npm dependencies.

When building from the official source tarballs which include pre-built frontend files, the frontend target will not be triggered, making it possible to build without Node.js and Internet connectivity.

Parallelism (make -j <num>) is not supported.

More info: https://docs.gitea.io/en-us/install-from-source/

Using

./gitea web

NOTE: If you're interested in using our APIs, we have experimental support with documentation.

Contributing

Expected workflow is: Fork -> Patch -> Push -> Pull Request

NOTES:

  1. YOU MUST READ THE CONTRIBUTORS GUIDE BEFORE STARTING TO WORK ON A PULL REQUEST.
  2. If you have found a vulnerability in the project, please write privately to security@gitea.io. Thanks!

Translating

Translations are done through Crowdin. If you want to translate to a new language ask one of the managers in the Crowdin project to add a new language there.

You can also just create an issue for adding a language or ask on discord on the #translation channel. If you need context or find some translation issues, you can leave a comment on the string or ask on Discord. For general translation questions there is a section in the docs. Currently a bit empty but we hope to fill it as questions pop up.

https://docs.gitea.io/en-us/contributing/translation-guidelines/

Crowdin

Further information

For more information and instructions about how to install Gitea, please look at our documentation. If you have questions that are not covered by the documentation, you can get in contact with us on our Discord server or create a post in the discourse forum.

We maintain a list of Gitea-related projects at gitea/awesome-gitea.

The Hugo-based documentation theme is hosted at gitea/theme.

The official Gitea CLI is developed at gitea/tea.

Authors

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

FAQ

How do you pronounce Gitea?

Gitea is pronounced /ɡɪti:/ as in "gi-tea" with a hard g.

Why is this not hosted on a Gitea instance?

We're working on it.

License

This project is licensed under the MIT License. See the LICENSE file for the full license text.

Screenshots

Looking for an overview of the interface? Check it out!

Dashboard User Profile Global Issues
Branches Web Editor Activity
New Migration Migrating Pull Request View
Pull Request Dark Diff Review Dark Diff Dark