* Update emoji dataset with skin tone variants
Since the format of emoji that support skin tone modifiers is predictable we can add different variants into our dataset when generating it so that we can match and properly style most skin tone variants of emoji. No real code change here other than what generates the dataset and the data itself.
* use escape unicode sequence in map
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Change language statistics to save size instead of percentage in database
Co-Authored-By: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Do not exclude if only language
* Fix edge cases with special langauges
Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
Unfortunately #11614 introduced a bug whereby the initial commit of a
repository could not be seen due to there being no parent commit to
create a clear diff from.
Here we create a diffstat from the difference between the parentless SHA and the SHA of the empty tree - a constant known to git. (With thanks to @L0veSunshine for informing me of this SHA)
Thanks to @a1012112796 for initial attempt to fix.
Fix#11650Closes#11674
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-Authored-By: L0veSunshine <xuan199651@gmail.com>
* [FEATURE] [API] Add Endpoint for Branch Creation
Issue: https://github.com/go-gitea/gitea/issues/11376
This commit introduces an API endpoint for branch creation.
The added route is POST /repos/{owner}/{repo}/branches.
A JSON with the name of the new branch and the name of the old branch is
required as parameters.
Signed-off-by: Terence Le Huu Phuong <terence@qwasar.io>
* Put all the logic into CreateBranch and removed CreateRepoBranch
* - Added the error ErrBranchDoesNotExist in error.go
- Made the CreateNewBranch function return an errBranchDoesNotExist error
when the OldBranch does not exist
- Made the CreateBranch API function checks that the repository is not
empty and that branch exists.
* - Added a resetFixtures helper function in integration_test.go to
fine-tune test env resetting
- Added api test for CreateBranch
- Used resetFixture instead of the more general prepareTestEnv in the
repo_branch_test CreateBranch tests
* Moved the resetFixtures call inside the loop for APICreateBranch function
* Put the prepareTestEnv back in repo_branch_test
* fix import order/sort api branch test
Co-authored-by: zeripath <art27@cantab.net>
When matching emoji, use a regex built from the data we have instead of something generic using unicode ranges. A generic regex can't tell the difference between two separate emoji next to each other or one emoji that is built out of two separate emoji next to each other.
This means that emoji that are next to each other without space in between will be now accurately spanned individually with proper title etc...
* add API specific InternalServerError()
Co-authored-by: zeripath <art27@cantab.net>
* return 500 error msg only if not Production mode
* Revert "return 500 error msg only if not Production mode"
This reverts commit 8467b2cee674ad205b452780ca88abb1b27643c8.
* InternalServerError
Co-authored-by: zeripath <art27@cantab.net>
When "Must Change Password" simplify the navbar header to only show the
signout button as all other links will redirect back. This prevents the
notifications icon from showing preventing initialization of the
event-source and hence preventing redirect_to being set, however in
addition do not set the redirect_to cookie if we are looking at the
/user/events page.
Fix#11554
Signed-off-by: Andrew Thornton <art27@cantab.net>
GH has different HardBreaks behaviour for markdown comments and documents.
Comments have hard breaks and documents have soft breaks - therefore Gitea's rendering will always be different from GH's if we only provide one setting.
Here we split the setting in to two - one for documents and one for comments and other things.
Signed-off-by: Andrew Thornton art27@cantab.net
Changes to index.js as per @silverwind
Co-authored-by: silverwind <me@silverwind.io>
Changes to docs as per @guillep2k
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Unfortunately when the virtual session is released it requires that the
real session does not exist. This worked fine when sessions were only
saved at the end of request/response cycle however, now sessions are
saved proactively this does not hold.
The result is a caught panic in the logs during every log-in. This
panic has no significant side-effects but should not occur.
This PR marks the virtual session as released when released and updates
it if the same session is released again.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Move serviceworker to workbox and fix SSE interference
Instead of statically hardcoding every frontend asset, this uses a
type-based approach to cache all js,css and manifest.json requests.
This also fixes the issue that the service worker was interfering with
EventSource because it was unconditionally handling all requests which
this new implementation doesn't.
Fixes: https://github.com/go-gitea/gitea/issues/11092
Fixes: https://github.com/go-gitea/gitea/issues/7372
* rethrow error instead of logging
* await .register
* Revert "rethrow error instead of logging"
This reverts commit 043162ba1f18b98a4bf9635959fd28d16e839fc5.
* improve comment
* remove JSRenderer
* add version-based cache invalidation
* refactor
* more refactor
* remove comment
* rename item to fit cache name
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
InternalTokens are fixed as alphanum strings therefore TrimSpace from these.
Also use isatty to not add a terminal newline when redirecting generate.
Fix#11498
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Refactor Cron and merge dashboard tasks
* Merge Cron and Dashboard tasks
* Make every cron task report a system notice on completion
* Refactor the creation of these tasks
* Ensure that execution counts of tasks is correct
* Allow cron tasks to be started from the cron page
* golangci-lint fixes
* Enforce that only one task with the same name can be registered
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix name check
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @guillep2k
* as per @lafriks
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add git.CommandContext variants
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Allow log.xxx.default to set logging settings for the default logger only
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/setting/log.go
* as per @silverwind add some documentation
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Handle panics in graceful goroutines
Adds a some deferred functions to handle panics in graceful goroutines
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Handle panic in webhook.Deliver
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Handle panic in mirror.syncMirror
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Properly generate ref URLs
Tags used to not generate correct URLs (src/branch/tags/1.0.0 instead of
src/tags/1.0.0).
Also cleans up some code around it with the created helper functions.
* Fix formatting and create migration
* Add copyright head to utils_test
* Use a raw query for the ref migration
* Remove semicolon
* Quote column and table names in migration SQL
* Change || to CONCAT, since MSSQL does not support ||
* Make migration engine aware
* Add missing import
* Move ref EndName and URL to the issue service
* Fix tests
* Add test for commit refs
* Update issue.go
* Use the right command for building JavaScript bundles
* Prepare for merge
* Check for refs/* before prepending in migration
* Update services/issue/issue_test.go
* Update modules/git/utils_test.go
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: techknowlogick <matti@mdranta.net>
* Allow compare page to look up base, head, own-fork, forkbase-of-head
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @guillep2k
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update routers/repo/compare.go
* as per @guillep2k
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Rationalise the names a little
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Rationalise the names a little (2)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix 500 with fork of fork
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent 500 on compare different trees
Signed-off-by: Andrew Thornton <art27@cantab.net>
* dotdotdot is perfectly valid in both usernames and repo names
Signed-off-by: Andrew Thornton <art27@cantab.net>
* ensure we can set the head and base repos too
Signed-off-by: Andrew Thornton <art27@cantab.net>
* ensure we can set the head and base repos too (2)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* only set headRepo == baseRepo if isSameRepo
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Initial support of end Range header option in lfs
* Allow end range option to be unspecified
* Declare toByte for consistency
* Factor out content encoding tests from doLfs
This is so Range tests could still use doLfs but without repeating
not related tests
* Add Range header test
* implemented extraHeader
* parametrized expectedStatus
* Add more test cases of Range header
* Fix S1030: should use resp.Body.String()
* Add more tests for edge cases
* Fix tests
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.
Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.
Unfortunately in `Workerpool` these were checked the wrong way round.
However, there were a few other places that were not being checked.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Manage port in submodule refurl
Fix#11304
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* URLJoin is causes a cyclic dependency and possibly isn't what what we want anyway
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Protect against leading .. in scp syntax
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
There is an unfortunate signature change with the api.Context
NotFound function; whereas the normal modules/context/Context
NotFound function requires an error or nil, the api.Context
variant will panic with an NPE if a nil is provided.
This PR will allow api.Context.NotFound to tolerate a being
passed a nil.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* routers: make /compare route available to unauthenticated users
Remove some bits of the compare interface if the user isn't signed in.
Notably, they don't need to see the "New Pull Request" button box nor the
hidden form that would fail to submit due to the POST request continuing to
require proper privileges.
Follow-up commits will improve the UI a bit around this, removing some
"Pull Request" verbiage in favor of "Compare."
* ui: home: show "compare" button for unauthenticated users
This change requires pulling in the BaseRepo unconditionally and
recording if the pull request is in-fact not allowed
(.PullRequestCtx.Allowed). If the user isn't allowed to create a pull
request, either because this isn't a fork or same-fork branch PRs aren't
allowed, then we'll name the button "Compare" instead of "Pull Request."
* ui: branch list: use the new Compare language when available
When viewing the branch listing as an unauthenticated user, you'll get
"Pull Request" buttons. use the new "Compare" verbiage instead, which
matches GitHub behavior when you can't issue a pull request from the
branches.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Add test
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Restore checkbox rendering and prevent poor sanitization of spans
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Also fix preview context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Also fix preview context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Make sure that sendmail processes register with the process manager
* Provide a timeout for these (initially of 5 minutes)
* Add configurable value and tie in to documentation
* Tie in to the admin config page.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Now that emojify.js has been removed, get rid of all instances of has-emoji class that was only used for that. Support for rendering shortcodes should remain in all of these places so it should still work the same.
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
In #9888, it was reported that my earlier pull request #9075 didn't quite function as expected. I was quite hopeful the `ValuesWithShadow()` worked as expected (and, I thought my testing showed it did) but I guess not. @zeripath proposed an alternative syntax which I like:
```ini
[markup.sanitizer.1]
ELEMENT=a
ALLOW_ATTR=target
REGEXP=something
[markup.sanitizer.2]
ELEMENT=a
ALLOW_ATTR=target
REGEXP=something
```
This was quite easy to adopt into the existing code. I've done so in a semi-backwards-compatible manner:
- The value from `.Value()` is used for each element.
- We parse `[markup.sanitizer]` and all `[markup.sanitizer.*]` sections and add them as rules.
This means that existing configs will load one rule (not all rules). It also means people can use string identifiers (`[markup.sanitiser.KaTeX]`) if they prefer, instead of numbered ones.
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Support unicode emojis and remove emojify.js
This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.
This works in a few ways:
First it adds emoji parsing support into gitea itself. This allows us to
* Render emojis from valid alias (😄)
* Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
* Easily allow for custom "emoji"
* Support all emoji rendering and features without javascript
* Uses plain unicode and lets the system render in appropriate emoji font
* Doesn't leave us relying on external sources for updates/fixes/features
That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)
For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.
The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.
I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.
I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.
Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.
Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130
* add new shared function emojiHTML
* don't increase emoji size in issue title
* Update templates/repo/issue/view_content/add_reaction.tmpl
Co-Authored-By: 6543 <6543@obermui.de>
* Support for emoji rendering in various templates
* Render code and review comments as they should be
* Better way to handle mail subjects
* insert unicode from tribute selection
* Add template helper for plain text when needed
* Use existing replace function I forgot about
* Don't include emoji greater than Unicode Version 12
Only include emoji and aliases in JSON
* Update build/generate-emoji.go
* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have
* final updates
* code review
* code review
* hard code gitea custom emoji to match previous behavior
* Update .eslintrc
Co-Authored-By: silverwind <me@silverwind.io>
* disable preempt
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
As seen in trouble shooting #11032 the new feature of Go 1.14 is causing several second delays in startup in certain situations. Debugging shows it spending several seconds handling SIGURG commands during init:
```
6922:04:51.984234 trace init() ./modules/queue/unique_queue_wrapped.go
remote: ) = 69 <0.000012>
remote: [pid 15984] 22:04:51 write(1, "\ttime taken: 236.761\302\265s\n\n", 25 time taken: 236.761µs
remote:
remote: ) = 25 <0.000011>
remote: [pid 15984] 22:04:51 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
remote: [pid 15984] 22:04:52 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
remote: [pid 15984] 22:04:52 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
```
This causes up to 20 seconds added to a push in some cases as it happens for each call of the gitea hook command. This is likely the cause of #10661 as well and would start to effect users once we release 1.12 which would be the first release compiled with Go 1.14. I suspect this is just a slight issue with the upstream implementatation as there have been a few very similar bugs fixed and reported:
https://github.com/golang/go/issues/37741https://github.com/golang/go/issues/37942
We should revisit this in the future and see if a newer version of Go has solved it, but for now disable this option in the environment that gitea hook runs in to avoid it.
* Prevent panic during wrappedConn close at hammertime
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/graceful/server.go
* Fix extraneous debug in goldmark.go
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix checkbox rendering
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Normalize checkbox rendering
Signed-off-by: Andrew Thornton <art27@cantab.net>
* set the checkboxes to readonly instead of disabled
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Add control for the rendering of the frontmatter
* Add control to include a TOC
* Add control to set language - allows control of ToC header and CJK glyph choice.
Signed-off-by: Andrew Thornton art27@cantab.net
* as title, do same changs on action view with #10737
* chage default icon from "invalid type" to "question" , because "invalid type" is not a meaningfull icon type
Signed-off-by: a1012112796 <1012112796@qq.com>
mrsdizzie
6543
Cirno the Strongest
Lauris BH
zeripath
Terence Le Huu Phuong
silverwind
赵智超
Sijmen Schoon
burbon
Kyle Evans
Alexander Scheel