banner_jpg
Username/Email: Password:
Forums

Manga Cover Database

You must be registered to post!
From User
Message Body
user avatar
Member

12:39 pm, Aug 29 2011
Posts: 298


Parts suggestion, request and announcement... Mainly directed at the staff.

A couple of weeks ago there were some requests (not on this site... I think) as to a scraper/plugin for manga, in particular for ComicRack (collection manager/reader, mainly focussed on western comics).

The first mayor obstacle we encountered was actually getting the information from somewhere, we were amazed that there really isn't any online database that is on the same level as we were used to with American and European comics (i.e. comicvine and bedetheque). Think per volume descriptions, ISBN numbers, covers, per volume ratings, etc. etc.
At most, there is some generic information about the series, but pretty much nothing about the actual volumes (without resorting to Amazon.co.jp).
Since I've always liked and used MangaUpdates as my main source of information anything Manga related (the fact it's quite large also helped) I've decided upon using this site as my source of info.

The next big obstacle was covers, since many collection managers are cover orientated (looks quite nice and is actually useful in recognizing manga) it would be nice to have a per-volume cover available (of actually decent quality)... Again, to my personal amazement, there's no such thing on the web (some, but only with a couple mangas and are broken/have ugly quality)... I've therefore decided to put some time into making my own.
Yes, I know manga is legion and quite a lot of mangas don't have covers (serialised)... For now, I'm just going to ignore these since there is no feasible way to link these series to which actual magazines they were released in (I have some magazine covers, but by themselves they're pretty useless).

Manga Cover Database
Example: Yotsubato!
Note: all manga information is retrieved from MU, everything is cached locally. No further contact with MU should be necessary. My site is mainly focussed on providing content for scrapers/scripts, without stressing MU server(s).
The interface isn't fancy and definitely not a wonder to behold, but it's functional and of secondary import.

Currently, I'm working on a way to keep the series information up-to-date. My idea is to weekly check the 'ongoing' series and decrease the update frequency whenever there have been no updates for a period of time... Shouldn't be a hard thing to do, but it's something that requires some thinking and a lot of waiting.

Another thing that I really want to add is a user system, currently the database 'only' contains about 350 series of which most are either still ongoing or missing one or two covers. Since there is only so much I can do by myself, I'd like to ask the community for some help... Which would require a working content submission system.


My questions are these:
1. Is there any help you can offer me (server-side) in keeping the series up to date?
I'd prefer (and I think so would you) to keep everything cached locally to reduce stress to your servers and decrease search/load times. Maybe via some sort of API/page that can list all changed series since/in the last request/week?
Ideas about other ways to approach this are also welcome (I currently try to keep cross-server interaction as low as possible, maybe not necessary?).

2. What can you recommend in ways of user system?
I'm not really interested in users per se, just a way to manage possible contributors and their contributions (ensure quality and prevent 'malicious' additions/removals). Maybe some ready made (and tested) system?

3. Is there any interest in maybe integrating this 'idea' into MU?
I.e. a cover wall.
The total filesize of all covers numbers into the GBs, but all (decently sized) thumbnails combined is considerably less (<30MB).
Of course, this would require a bit more series with covers.

This one goes out to all developers:
4. Anyone interested in actually making some plug-ins?
Think of programs like Collectorz, ComicRack, GCstar, etc.
I've tried to do most of the work in preparing the data (if you have requests about delivery please contact me).

P.S.
I've thought about placing a MU banner on my site (giving proper credit), but decided against it before having your explicit approval, until that time I don't want to make myself seen as approved or condoned by MangaUpdates. If you would like to have one placed, feel free to contact me.

Last edited by Joentjuh at 12:48 pm, Aug 29 2011

________________
Who they, what are, and why?
- Manga Cover Database -
user avatar
Member

11:48 am, Sep 5 2011
Posts: 298


No one interested? (or is it too much text? bigrazz)

________________
Who they, what are, and why?
- Manga Cover Database -
Post #497477
user avatar
Member

7:00 pm, Sep 22 2011
Posts: 437


So, you want to make a database of volume covers and also display series information taken from MU? I have mixed feelings about this, and I don't seem to be able to string them together into a coherent response, so I'm just going to make a bulleted list.

-It seems like you want to make a copy of MU, but with covers of each volume included. If so, the info is the same as MU, so why bother taking it to a new location? Why not just make a database of covers and link/refer users to MU for any other info?
Edit: Nevermind, that comment was based on the version of your site that I looked at a week or so ago. I looked at your site again, and now it seems it's changed to have just images and series names.

-Having a cover gallery for each series incorporated in MU sounds lovely, especially if someone has already collected all the images already, but I'm pretty sure it's not going to happen. It has been suggested on the forums here, but as far as I know it would take up too much space (although that's just my guess). Also we are short on coding staff, so... at the least, not in the near future.

-I've spent a lot of time on the web looking up covers for series, since it's one of my favorite things to do here. I've found that actually there are a lot of good resources if you're looking for volume covers. None of them are perfect, but among shopping sites and book review sites and, if necessary, google and auction sites, you can usually (Somewhere around 80 to 90% of the time, I think... though I'm bad at estimating) find a cover for a series. (If you're curious, PM me and I'll gladly tell you everything I know about finding cover pictures, the sites that I use, etc.) Also, as time goes on, the sites that have images keep getting better, having larger images and more series... It'd be really hard for you to even come close to them. BUT, none of these sites are in English, and thus they are hard to find and even harder to use for manga fans who don't know any Japanese. For popular series it's not a problem, but outside of those, I think finding a cover picture without knowing Japanese would be difficult. A site in English, with a sort of gallery like you have of large, high quality cover images for all volumes of series, naturally sounds wonderful.

-I especially like the way you've laid out the images. Having a sort of wall with all the images is so much nicer than most places, which have little thumbnails, and you have to click on each volume to see a decent-sized version of it (though I understand it makes pages load faster, which is important for busy shopping sites), or have each volume accompanied with text (descriptions, etc.) so that you have to scroll down a lot, when I don't even care about the text to begin with. (I mean, make the text available if I really want to see it, but make it a link or something so that it's out of my way, because I usually don't.) The way the picture pops up as you mouse over the name of a series on the index is also nice--it keeps the list compact, but you don't have to waste time clicking on the link to get to see the picture, going back to the index, clicking on the next one, etc.
-About the volumes, how do you plan do deal with multiple editions? There'll be the Japanese edition and usually the English one, and there may be other languages as well, and then there are special editions--bunkoban, kanzenban, shinsouban, etc.--which usually have different covers. Also there can be complications where one publisher prints say volumes 1-5 and then stops, and then another publisher takes the project and reprints volumes 1-5 under their cover design and then continues the series from there.

-Sorry I don't know much of anything about web programming, plug-ins, readers, etc., so I can't answer any of your questions.

Last edited by lynira at 7:20 pm, Sep 22 2011

________________
Manga Cover Database
Post #497492
user avatar
Site Admin

9:54 pm, Sep 22 2011
Posts: 2275


lynira...

We once had a link to older image uploads on the site; however, we can't have nice things, because there were images that were not censored in some series' caches. So, it was removed.

________________
"Officially, this machine doesn't exist, you didn't get it from me,
and I don't know you. Make sure it doesn't leave the building."
Post #497525 - Reply to (#497492) by Toto
user avatar
Member

5:17 am, Sep 23 2011
Posts: 298


Quote from lynira
- long post -

Someone was already kind enough to point out that simple fact to me by email. I had/have no intention of creating a clone of MU and have therefore decided to remove all superfluous information and replaced it with a hard-link to the correct MU page and a 'Feeling lucky' link to MyAnimeList (for those who prefer that one)... This actually made things easier for me (less data to sync).

I'm aware there are many sources for covers, most noteworthy Amazon, but these covers are often of atrocious quality and hard to find. Many scanlators also offer front covers for series they scanlate (at project pages and such), whilst often being of decent enough quality for use in collection managers, they still are pretty scattered and they only account for a few series.
I'm simply trying to gather these covers and make them more accessible for all your collection manager uses (especially for those, as you have rightly indicated, not familiar with the kana).
My solution might not be perfect, but I'm hoping it's a good start to 'organizing' the manga world and a stepping stone for other future projects... Can't think of anything particularly useful right now, but isn't that how many things start?

For the amount of volumes for a series, I take the 'highest' number listed in the "Status in Country of Origin" field and if that field also contains the text 'Completed' I mark that series as completed as well... Maybe not the best way to handle this but it suffices for now.
Ongoing series are only updated after 14 days have passed since the previous update and only when a user actually visits that page (no visit = no update).

About the multiple editions, currently I simply show the 'primary' cover for that volume and via the browser you can see if there are other versions available. (i.e. _a and _b).
If there's a compiled volume (i.e. one volumes contains three of the original releases) I simply use that volume number and put it in the alternate covers.
I don't keep track of which is which, I initially tried to do that but it proved way to much work for something I don't particularly care about. Back covers can often be found via the browser button if you wish for ISBN numbers and such, I'd love to have those number play a more important role in the database but I don't see it happening in the near future.

This could change with some help of other developers/maintainers, but until then it's not going to change (something many a staff of online databases should be familiar with).
I do am going to try and integrate those others covers a bit more though, not sure how I'm going to do that yet...
I'm also thinking about creating some abstraction for the actual covers themselves, creating an API for this is definitely on the ToDo list... though this would require some solid brainstorming.

Quote from Toto
lynira...

We once had a link to older image uploads on the site; however, we can't have nice things, because there were images that were not censored in some series' caches. So, it was removed.


This is one of the problems I'm facing, currently I allow covers for all kinds of manga under the condition that it has been published in stand-alone volumes (so no magazine covers and preferably no doujinshi.. though I might reconsider with a good enough argument or some support on that front).

This also includes content that is not suitable for all age groups, whilst I personally don't read yaoi and hentai (with the exception of maybe a small few borderline) I can't really be bothered to process those covers.
If users decide to submit those to me, I'll gladly host it... but it's not something I'm going to do myself.

I'm not even going to bother with censoring and such, series are listed by name and if MU has labelled the series as Hentai (amongst other similar labels) I'll simply put up a flag... Maybe I'll even throw in a filter as an extra.

________________
Who they, what are, and why?
- Manga Cover Database -
Post #497570 - Reply to (#497492) by Toto
user avatar
Member

11:20 am, Sep 23 2011
Posts: 437


Quote from Toto
lynira...

We once had a link to older image uploads on the site; however, we can't have nice things, because there were images that were not censored in some series' caches. So, it was removed.

I remember seeing that for a day or two, then yeah, it disappeared. I could go through the caches and either censor the images that need it or delete them... Well, that's what I was going to say, but then again... the majority of those images are going to be small (the old max image size or smaller), since we only increased the max size recently, and also some of the old images for series are wrong (someone uploaded a cover for a different series by mistake, etc.), or they used some other artwork that's not a cover, so they're not that great after all... :/ But it'd be nicer than nothing.

Quote from Joentjuh
I'm aware there are many sources for covers, most noteworthy Amazon, but these covers are often of atrocious quality and hard to find.
Actually I don't like to use Amazon for cover images... they may have some very high quality covers where you can zoom in really close and the quality's still good, but for them, the interface won't let you save the whole image, so it's a hassle. Also Amazon doesn't have as many series as other sites. On image quality, though, a lot of places do have either pictures that are too small, or low resolution, or both--like 7andy, a site which I liked to use a lot in the past: their selection is great, but the images are usually small (under 350 pixels high) and sometimes low res as well. Nowadays, the best site I know of is called booklog; they have a big selection (smaller than 7andy's, though), but usually their images are 500 pixels high and good resolution. If you want images larger than that, though, then you're in trouble... I doubt you can find any site that has more than several hundred series like that whose images are easy to get (sites that have a tachiyomi feature, like Amazon does, give you a really great cover image, but they're a pain to get).

Last edited by lynira at 11:43 am, Sep 23 2011

________________
Manga Cover Database
user avatar
Member

11:23 am, Oct 16 2011
Posts: 298


Today I did a big revamp of the interface, navigation should be a lot clearer now and the different sections are now a bit more integrated.

I've replaced the MangaUpdates linkage button with a custom made image, if it's not to your liking (or have a better one available), let me know. Also, if you want more linkage, I'd be willing to oblige.
User Posted Image

I've also created a quick and dirty prototype of the Bookcase view.
For this feature, I'm making use of the public 'complete' list of a given user (at the moment only via userID).
Probably won't really continue working on this before the cover database has had some time to grow, but for now I'd love some feedback, ideas, or suggestions about what could be smile .

I also have a question: Is it possible to search for members? Preferably via some sort of 'feeling lucky' system.
This would make it possible for users to enter their username instead of userid in the bookcase generation form (without me needing to build some binary-search scraper - possibly generating much traffic).
The other option would be having the user make a manual export of a list and upload it to my site... which would requires more steps user-side and therefore pretty much a non-option.

Last edited by Joentjuh at 3:18 pm, Nov 19 2011

________________
Who they, what are, and why?
- Manga Cover Database -
Post #501515
Member

11:31 am, Oct 16 2011
Posts: 7


Interesting project, thumbs up.

user avatar
Member

8:13 am, Oct 23 2011
Posts: 298


Okay next step:
I've been able to successfully extract barcodes from nearly all covers that feature one (which is just somewhat less than half of the total). You can now search for these volume covers by barcode.

What I've not yet been able to achieve is automated scraping of volume information from 3rd party sources by ISBN number (EAN13), I had Amazon.co.jp in mind for this.
This way I would be able to also display things like the release date, publisher information, and book dimensions.

Anyone here who could offer his/her expertise on that subject?
Example: Yotsubato! Volume 1

Next up: More covers... I think. I've spend way too much time on coding lately.

________________
Who they, what are, and why?
- Manga Cover Database -
Post #514599
Member

2:13 am, Dec 28 2011
Posts: 2


Joentjuh, I like the idea of harnessing collective interests and centralizing efforts to build up a comprehensive cover database.  I've not looked around the Cover Database just yet, or read all of your comments on it, but I'll be poking around in the future.

I strongly recommend avoiding re-invention or competition wherever possible, though it's correct that the data has to initially come from somewhere, and that you should never over burden external resources. However, sometimes there will indeed be some overlapping. Looks as if we're in agreement. Collecting once from amazon or other sources is certainly one thing, though automated collection from other niche community resources must be approached carefully. Linking out and providing users with new services, and providing resource providers with new, motivated and involved users, should almost always be welcome. wink

Quote
What I've not yet been able to achieve is automated scraping of volume information from 3rd party sources by ISBN number (EAN13), I had Amazon.co.jp in mind for this. This way I would be able to also display things like the release date, publisher information, and book dimensions.

I've got a fair amount of data aggregation and screen scraping experience, so I may be able to help brainstorm approaches or hack up some tools or services. Though, first steps would be to determine just how consistent and comprehensive that data is. The other question I might propose - is it better to provide conveniences which enable users to submit this data (many different 'smart' but guided collection points); or to instead to automate the collection process by dedicating a single 'dumb' entity to this task.

[ On methods of building up the database.. ]

Surely, you could pre-load with data collected from an automated source, but then it may need to be later verified and scrubbed by someone.** Also, would you cache this volume data, and for how long? Or would you collect once and then not benefit from admins updating a source such as amazon (and instead rely on updates from an internal Cover Database or MU/etc community) ?
** (Note: Pre-loading on demand in the case of browsing, and not just when adding/editing entries, would make regulating hit's to external resources difficult.)

I'd view the automated 'dumb' collector as a tool to aid users interested in maintaining 'the collection'. While other tools might be designed to aid users in cross-loading data for specific titles in their collections as they come across them. ie. User looks at the entry provided by your service and decides that specific title lacks data or is incorrect, and that he/she wants the new data to be available through your (more convenient) interface.

Also, consider what's more frustrating.. no data, or lots of incorrect data? Would it be enough to flag data collected automagically so that users don't think the 'worthless' community is uploading lots of incorrect data? Should that automatically collected data be visible to regular users without obvious indication that it may be incorrect and that they can help by verifying it? Should we trust it as 'more-likely' correct, or drop priority on verifying it, if a given number of users have viewed it and not reported anything as incorrect? Just a few things to consider that came to mind.

[ On Providing Conveniences to increase Collaboration.. ]

Ideally, if users were offered some tools to help them maintain their collections, such that they could spend time providing and verifying data for media that matters to them; collectively everyone will build something which individually would cost too much. The more eyes, and the more often you have them on the data, the more accurate it will become as a whole. Again, that's probably obvious, but that's to suggest that API's and convenient interfaces/tools for interacting with the Cover Database and MU would certainly bring in more users, increase contributions by existing users, and further encourage collective management of their own media. (One piece to this you've already suggested a path for in developing plugins for the external clients you've mentioned, though I'm not yet familiar with them.)

Quote
I've also created a quick and dirty prototype of the Bookcase view.
For this feature, I'm making use of the public 'complete' list of a given user (at the moment only via userID).

Good idea. Not only does this provide users with direction and motivation of where they could quickly and easily contribute by dipping into their collections for cover art (and passive verification of other metadata); but it also doubles as a convenient way for them to collect and generate thumbnails for their own purposes.


I could be interested in helping out, by collecting data, or creating scripts/tools for aiding in collection and management of it. Though, I'm currently here because I'm surveying the resources available for tracking completion, progress, and the current state of my personal manga-reader profile, as well as for keeping up to date on releases and new titles by artists/authors/scan-translation-groups.  I've been using Manga Updates from the sidelines for a while, but have only just now signed up for an account.

How's progress, what are your ultimate goals, and what services or interfaces are you interested in providing, and for what audiences? Any rough answers would be fine. I'm just trying to get a feel for what you'd like to accomplish beyond maintaining a collection of covers. bigrazz


user avatar
Member

11:11 am, Dec 28 2011
Posts: 298


Okay, it's been a while since I last post here and many things have changed (also, I had totally forgotten about this topic), so a "little" update is in order. (read: wall of text \o/)

First of all, I'm aware my site is lacking and leaves much room for improvement, still with the limited amount of time and manpower (only me), I don't consider the current result bad... Especially since this site has been build without any real design beforehand and most features were added or changed on the spot (the advantage of having full control)... Against all odds, I've been able to keep the code relatively clean and structured...
There are a few things that really ought to be added (like a proper credits page) but haven't gotten round to doing yet... I hope to address some of these things the coming days (having just finished reading the 12 kingdoms novels, I have some time on my hands bigrazz ).

= Goals =

Collect and share manga covers for the community
- It amazed me that something like this didn't already exists, when it does for pretty much anything else... Sure some sites host covers, but only a select few and never in a structured way or in low quality... So far I've tried to stick with the "original" covers (if I was able to find them).
And collect volume specific information in a singular place, for easy access
- In other words, add a bit of structure and allow for third-party access.

= On scraping and caching =

At the moment I use mangaupdates as a primary source of information and mangas are identified by mangaupdates IDs, meaning that my database depends on mangaupdates never changing manga IDs...
Manga information is only updated when content on my side is changed, meaning when a cover is added or changed... The downside is that it's likely information becomes outdated, but only really the number of released volumes is affected.
As mentioned before and I hope it's quite clear, the purpose of the site is to provide covers and optionally volume specific information, not series information. For anything more than covers or volume information, visitors are directed to either MangaUpdates or MyAnimeList.

Furthermore, ALL information collected from any remote site is cached locally (which could be more than what I store in the actual database, and consequently show on the website)... This because I scrape more information than I need or use at the moment, and because I like having a buffer (scraping from remote sites is SLOW, especially when you have over a thousand series and over 10.000 volumes, I use random delays between requests varying from 2 to 5 seconds... because I'm a nice guy).
That, and all "raw" cache combined amounts to only 45MB.. which is next to nothing.

The procedure is as follows:
1. I manually place covers in the correct directories (I personally look up each new series on mangaupdates and assign the correct ID)
2. I run my updater script:
- MangaUpdates is scraped for all series with new, or altered, covers
- MyAnimeList is scraped for newly added series only (to further extend series titles)
- All backside covers are automatically scanned for readable barcodes, ISBN are stored -- sadly still fails to read a barcode every now and then, and OCR has to large an error margin to be reliable
- Amazon.co.jp is scraped for volume specific information
- If not found on amazon, or certain fields are missing, kinokuniya is also scraped

= User system =

I've not started on building, or implementing, a user system and I'm not sure if I'm even going to do that. The website construction as it is now is optimised primarily for speed (loads of caching is being done), If a user system was to be added, this would probably require either dynamic content -> javascript/ajax or partial caching. I'm not that good with javascript and not really a fan of it either (I'm old-school that way). help here is much appreciated

As to the point of user maintained content, this is indeed something that could prove very useful and definitely something to be considered, but this also raises the points of security and validity. Whilst the security is very doable with my current skill set and experience (though not something I'm looking forward to), the validation of user entered content is something that would need to be thought out thoroughly.
Would it be better to have some sort of system in place that requires to approval of x visitors before changes are mode official, or should I implement a moderator system... and how could I best do this (I foresee loads upon loads of code and test-cases).
I'm not even sure there are enough people willing to contribute to make it worth my while to implement a "complex" user system, I might at the moment have about 150-200 unique visitors a day, I suspect only a few of them are willing to register, let alone go through the trouble of mutating records.
I could start with building something basic, but I'd rather do this right from the start than to improvise as I go along (since re-factoring is harder with a user system, and code complexity is increased significantly).
Would be nice to bounce some ideas around on this with someone...

= Bookcase =

The bookcase also has mature a bit, though it's by no means finished are anywhere close to it. For now MangaUpdates should be searchable by username and a very basic list parser is in place.... I'm not really sure where I want to go with this and how I'm going to do it, especially since I'm limited to how MU presents user lists and the inherent problem with "reading state" (up to which volumes has been read, assuming the user didn't skip any volumes). Ideas and/or examples anyone?

= API =

I do have a API in place (still very much in it's alpha stage) as requested by ComicRack. Still waiting on feedback before I continue with this side-project.
All content is returned in JSON format to allow easy processing by scripts, plug-ins, or for dynamic content... Might be tempted to add more output formats like xml,yaml,php (dump),csv,etc. (not that hard to do).

= Current progress =

I've thought about adding links to online manga readers... But so far I'm not sure this is something worth adding or even which site(s) to use.
The other thing I'm currently trying to work out is how to separate volumes (different prints).
Example: Rurouni Kenshin, I have 22 volume covers (which is the complete series in kanzenban), but there are multiple releases of this series each with a differing number of volumes.
I have some ideas on this, but no easy way of implementing since I largely rely on automatic information gathering. I simply have no information about which cover belongs to which type of releases (besides manual confirmation, which I'm not even going to consider).
So far, the series with multiple releases are a minority and I have deemed in not important to invest much time in... Anyone?

I'm kind of proud of my current search engine, though still being quite basic, and would love some ideas on how to further refine the results. As you might have noticed, currently the results are ordered by how well they match the search terms (or at least they should be). Partial word matches rank lower than full matches, and titles with the highest "match" percentage are put on top. The less extraneous words/characters in the title, the better the match.
Meaning that "dragon eye" ranks higher than "dragon ball" when searching for "dragon" because "dragon ball" contains more extraneous characters.

As always, any help given is welcome. Not just in content, but also in scripts or even general suggestions.

PS:
Thanks for showing an interest, it's what keeps me going smile .
The effect of even something as simple as a thanks is not to be underestimated.

Last edited by Joentjuh at 11:43 am, Dec 28 2011

________________
Who they, what are, and why?
- Manga Cover Database -
Post #514895
Member

1:52 am, Dec 30 2011
Posts: 2


Just a quick thought about the bookcase...

For anything in a user's list which lacks a cover, you might display it with an indication that it needs a cover. It might be a solid color that represents a missing spine, or even just a missing front/back cover. These volumes could be place in-line with the rest of their collection, or instead on a separate shelf. If in-line, maybe use transparency or wire frame outline for the volumes that lack covers. At the very least, if it isn't done visually, some statistics on how much of their collection is missing covers could be provided, along with a link to a page listing links to said titles.

Or, as a small bit of motivation to provide covers, if any of a user's volumes lack a cover you could pull all of the volumes for that series and place them on a separate shelf. Then, once covers are provided, they'll move up into the rest of the collection. (Though, in-line is probably better form.)

Wall of text follows... I'll try to take some of this over to your form in the future, and keep shorter or MU relevant comments here.

[ Personal Collection Tracking & Detailed Reading states ]

Quote
.. I'm not really sure where I want to go with this and how I'm going to do it, especially since I'm limited to how MU presents user lists and the inherent problem with "reading state" (up to which volumes has been read, assuming the user didn't skip any volumes). Ideas and/or examples anyone?

Yes, unless MU's reading state end's up being expanded upon, there will certainly be trouble. Currently, the reading state is just that, it's less about which volumes you have, and more about where you left off. For example, I own volumes 1, 4 & 5 of Someday's Dreamers Spelbound. If I'd read them, there's currently no way to record that status without loss of information. Granted, non-contiguous/non-linear reading states aren't as common, so it's understandable that from a reading status perspective there might be less of a priority on implementing support. If the goal was to also track possession, even if only by generic volume/chapter, which I'm quite interested in, then tracking fragmented statuses might be more reasonable to implement.

[ Mobile Personal Manga Collection Browser, anyone? ]

.. for Android / iPhone-pad-a-ma-jig. (aka.. distractions and motives for implementation of some of the necessary features)

Photos of your physical book shelf can only take you so far. Photos also require you to remember to keep them up-to-date so that you know what to buy when you're out. I'd gladly implement a portable bookcase browser for Android/iOS just so I could verify which volumes I was missing when I'm standing in front of a table at a convention or a newly discovered used book store. That would actually be simple with convenient API's to MU and your Cover Database, which already has a spiffy Bookcase view for anyone also wanting a visual representation. wink How many times have I been out and not sure of which volumes I'm missing... well, enough to have the required amount of annoyance fuel to consider implementing such an application.

Other perks might be:
- Marking what you've read, either online/offline, though offline would require the user to resolve their own transaction conflicts.
- Maintaining lists, specifically wish-lists. Scan the barcode/isbn, look up reviews etc, add to wishlist?
- and maybe some other convenient features I'll not mention here. roll

The above was just some thoughts that aren't actually too far fetched. Even though most reading now-a-days happens on a computer connected to the internet and thus reduce the chance anyone might need to maintain a list on the go... the ability to check your collection conveniently from your phone and manage wish-lists would actually be pretty useful to some. I'd be more concerned about implementing the features/services for such a system before a mobile app, but it's really only a few more steps after the infrastructure is laid out.


Quote
... Might be tempted to add more output formats like xml,yaml,php (dump),csv,etc. (not that hard to do).

Yeah, stick with json for now. It should also work nicely for if and when some ajax is added. yaml is nice, but it'd probably get less use. I'd say getting the API to a mature state is more important than adding xml support, at least until someone else needs it.

[ 'Automating' Edition tracking/discovery ]

Quote
The other thing I'm currently trying to work out is how to separate volumes (different prints).
Example: Rurouni Kenshin, I have 22 volume covers (which is the complete series in kanzenban), but there are multiple releases of this series each with a differing number of volumes.

I've recently been giving this some thought. I'll try to work out potential schema that would work well for a relational database.

Quote
... no easy way of implementing since I largely rely on automatic information gathering. I simply have no information about which cover belongs to which type of releases (besides manual confirmation, which I'm not even going to consider).

Let me begin by stating that it may not be as common, but if you later find there's interest in tracking publications outside Japan, you will likely want a schema and some tools to help with managing this; however, you are correct that tracking things like Japanese releases in magazines/anthologies (beyond the current scope), but also multiple printings, editions, or 'omnibus' releases would not be worth it when there might only be a handful of moderators/contributors, especially when trying to sift through automated results. Even so, you might significantly reduce the effort required rather than looking for a completely automated solution. Below is one such option, though at the current state of contributions and users it may or may not be worth the time to implement.

You could use image matching algorithms to compare your newly acquired cover with the thumbnails/samples from amazon/kino/etc. For the initial creation of an 'edition' you might look at your list of matches manually, and then once linked it would be easier to have the others fall into place. Honestly, manual checking will be the only way to ensure a reasonable level of validity, but you could still automate the discovery process and significantly reduce manual inspection time.

Consider this use case: You are provided a cover, which you know is a volume of Yotsuba. Better yet, you probably were given or know the volume number. You could scrape results for Yotsuba, then compare the resulting thumbnails/covers to your cover in question. If you find a potential match, you may now have linked it to information that will make the edition uniquely identifiable. You can look through the potential matches and if you can confirm a correct match, you may have found a link to an existing edition in your database, or instead discovered a new edition. You could then manually connect/import the necessary information.

On the technical side, you could use your own implementation of common image matching algorithms, or be lazy and upload the images of both your new cover and any potential records from your search of 'Yotsuba volume X', to Tin-Eye or a similar service. Biggest issue is having cover samples that are large enough, and that don't contain watermarks that are enough to throw off results. Again, this is a bit high-hoped, but if such a script could allow for significantly reducing the number of records you have to manually look through, or if it could bring to your attention things you wouldn't have to actively look for, then it might ease the process. (Also, if the image-matching was already in-place, you could do a google-goggles like search from a computer/phone, even without a matching/readable barcode.) The real question is - would the data you currently get from scraping be enough to identify editions by publication date or other info?

Quote
So far, the series with multiple releases are a minority and I have deemed it not important to invest much time in... Anyone?
Yeah, basically.. but if anything else comes to mind on how one might simplify things while aiming to stay automated, I'll be sure to mention it. Though, as mentioned, I'll gladly put some effort into how one might 'efficiently' store data for multiple editions.


So I've rambled out a bunch of thoughts on uses of the database and the information it would contain, and it's use in combination with MU or other similar services. I don't mean to suggest taking the project in directions you're not interested in, or directions that may not have much use/marketability, but I wanted to provide some interesting ideas which might further motivate, or just provide guidance on what others might use it for while moving forward. My strongest interests are in a service providing more detailed collection tracking, which is beyond the scope of the cover/volume database. That functionality would be best provided by improved features for MU, or by another service entirely; however, such services certainly do best to build upon and encourage expansion of the Cover Database.

I've not forgotten about the question of implementing a user system to allow management of the moderation and verification of the data. I'd also need to think some about this so that it might be designed with forethought, but not terribly complicated at first.

Last edited by spiders at 2:16 am, Dec 30 2011

Post #517042 - Reply to (#514895) by spiders
user avatar
Member

6:00 am, Jan 13 2012
Posts: 298


Quote from spiders
[wall of text]


I don't mind taking this project into directions I personally don't consider important, as long as there are people willing to help with this (on at least one or more levels of development).
Some of the points raised, such as a mobile application, also have crossed my mind before. Lacking time, experience, a development platform, and specifications I have not been able to start with this (very low priority at the moment).
As said before, here and on other forums, if someone else is willing to invest some time into this, I don't mind exchanging ideas and designs and tailor the API to their needs.

I've already done some preliminary database (re)designs to better store multiple releases, add user support, and refine it a bit overall (It's very basic and not really up to my own standards)... But I welcome any and all ideas.

I've also added a dummy spine cover for missing volumes (of series that are in the database) to the bookcase, so far only tested on (latest) Firefox and Chromium (though with the exception of pre-7 IE and crappy render engines such as that of the PS3) it should work.
I could even present the users with a list of missing series, though I'd only be able to tell them the Manga Updates ID and the "primary" title.
Not sure if it's a good idea to automatically scrape these series and put it in the cache. The advantage would be that it gives me a better impression of which series have a higher "collecting" priority and that I can better inform the user of which series are missing and how many volumes these have (i.e. dummy covers in bookcase)...
The downside is that scraping takes time, these series would be put in the "to scrape" queue that runs in the background, whilst this delay is fine with 2 or 3 series, with over a 100 series it becomes a bit harder to ignore. I'd also be storing a lot more information than necessary, whilst storage space is not an issue, having series without any covers stored in the database would be kind of pointless and might even become a hindrance.

Note that I only story information about series to which I have at least one cover and even then only what I require for sorting/searching purposes and to generate "status" information. (so no descriptions, comments/reviews or ratings)

Last edited by Joentjuh at 8:36 am, Jan 13 2012

________________
Who they, what are, and why?
- Manga Cover Database -
Member

10:15 pm, Dec 7 2012
Posts: 42


I've thought about this too.
It'll be pretty cool to have a shelf like that provided by anobii
http://www.lifehack.org/articles/technology/review-anobii-for-iph one.html
But it'll be manga specific, together with the recommendations and discussion, it seems good.


You must be registered to post!