Overview

Scope

DMLR’s objective is to publish high-quality scholarly articles focused on the data aspect of machine learning research. Broadly defined, DMLR covers but not limited to the following topics:
  • Datasets for machine learning research
  • Benchmarks for machine learning research (collections of datasets with particular aims)
  • Benchmarking tools and methods
  • Novel methods for data exploration, data preprocessing, data analysis, or including the knowledge of domain experts about the data
  • Data generators and reinforcement learning environments
  • Methodology and empirical evaluation of data collection processes, data generation, data labeling, data augmentation processes, generalizability of datasets, feature representations, text generation models, and image generation models
  • Societal and ethical studies around creation and uses of data
  • Fundamental contributions (theoretical or empirical) on various aspects of data quality, including data bias, variance, uncertainty and their influence on ML
  • Algorithms for data cleaning, acquisition, quality evaluation, and alignment for ML
  • Prompt design and creation for generative and foundational models
  • Experimental design, registered experiments, methodology of empirical evaluations, including design of competitions and benchmarks
  • Frameworks for responsible dataset development, audits of existing datasets, identifying significant problems with existing datasets and their use
  • Systematic analyses of existing systems on novel datasets or benchmarks that yield important new insight.

Content and Originality

DMLR welcomes both original contributions and expanded versions of papers that were previously published at conferences or workshops. In cases where a submitted paper is an expanded version of a prior publication, it is permissible to reuse the written content, figures, or findings from the earlier work as long as the DMLR submission enhances the previous publication by introducing at least more than 30% new content. Additionally, authors are required to disclose this information on the Openreview submission form and provide a link to the published work as a reference. DMLR also allows authors to submit their work concurrently to other non-archival venues or preprint servers, such as arXiv and bioRxiv, but not to another archival venue.

Format

DMLR accepts submissions via OpenReview, a transparent and open platform for scholarly peer review. Submissions should be generated as a PDF file using the DMLR LaTeX style template. The submission file can contain an Appendix (for proofs, derivations, or additional results that complement the main content) located after the references. Failure to adhere to the style format may result in desk rejection.

DMLR encourages authors to include supplementary material that directly supports the content of their submission. This may be data, source code, or explanatory videos, among other possibilities. Authors might want to look at Joelle Pineau's useful checklist about reproducibility in machine learning.

In particular, submissions introducing new datasets must follow the dataset submission guidelines of NeurIPS Datasets and Benchmarks by including the following in the supplementary materials (as a separate PDF):
  • Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, data cards, and accountability frameworks.
  • URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers.
  • Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license.
  • Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance.
Submissions are not constrained by a page limit; however, authors should be mindful that the review process for lengthy papers may require an extended period. The decision to consult the appendix or supplementary material is left to the discretion of the reviewers.

Broader Impact and Ethics Guidelines

Authors must include a Broader Impact Statement, which should provide a concise, tangible portrayal of both the potential positive and negative societal consequences of their work. We recommend authors to read A Guide to Writing an Impact Statement before they start their impact statement. Submissions must comply with the DMLR Code of Conduct and NeurIPS Code of Ethics. As such, we advise authors to read it for a comprehensive examination of potential risks and strategies for mitigation.

Submission Procedure

To submit a paper, please
  1. Prepare your submission in PDF format using the DMLR LaTeX style template (if the submission contains multiple files such as datasets and source code, please create an archive in tar or zip format)
  2. Go to OpenReview and log in
  3. Select the "submit manuscript" link across the top and upload your manuscript into the system
  4. Fill in the OpenReview form answering the following questions:
    • Is your submission an expanded version of any of your previously published papers?
      • If yes, please describe what additions have been made to the submitted DMLR paper
    • Confirm that all co-authors are aware of the current submission and consent to its review by DMLR
    • Declare possible conflicts of interest; in particular, name all action editors that have recently collaborated with authors of the submission
    • List of keywords from the article
You can monitor your submission by logging into OpenReview and going to “Your Active Consoles”.

Conflicts of interest

(Based on TMLR)

All authors are required to have an OpenReview profile and identify domain and personal conflicts. Domain conflicts should be indicated in the "Education & Career History" section, to declare institutions that you have a conflict of interest within the preceding three years, at minimum. Please exercise careful consideration when declaring domain conflicts, reserving their use for instances where a genuine conflict truly exists. It is strictly forbidden to falsely claim a conflict in an attempt to prevent reviewers associated with a specific institution.

For recent or ongoing collaborations of all sorts, you should generally use personal conflicts, which are recorded in the “Advisors, Relations & Conflicts” section. The following constitutes a personal conflict (adapted from NeurIPS and TMLR guidelines):
  • Family or close personal relationship
  • Ph.D. advisee/advisor relationship
  • Current, frequent, or recent collaboration (where recent means within the past three years)
There could be instances where a personal conflict exists, not falling within the defined categories mentioned above, but still holding the potential to substantially undermine the impartiality of the review procedure. In such situations, you have the option to mark such a conflict by adjusting its visibility settings within your OpenReview profile. Should an action editor or the editor-in-chief of DMLR have any reservations about the authenticity of such a conflict, they may confidentially inquire into it further.

Review process

DMLR employs a single-blind review process with open reviewing. This open reviewing model, which makes papers, reviews, and discussions accessible to the public upon the decision, provides transparency to the review process, fosters higher-quality and more constructive reviews, and facilitates a communication line between authors and reviewers. DMLR’s review process follows that of TMLR and can be summarized in the subsequent phases:

  1. Submission. A submission is made on OpenReview
  2. Reviewing. Within two-week timeframe, your submission will be pre-screened by the Editors-in-Chief (EIC), and they will assign an Action Editor (AE) to the submission. The AE will conduct an initial assessment of the submission. In cases where the submission falls outside the scope of DMLR, violates the required format, or clearly falls below the DMLR’s quality standards, the AE may opt to reject it. Alternatively, if the submission aligns with DMLR's guidelines, the AE will appoint three reviewers, beginning the public review process. Reviewers are asked to submit their reviews in four weeks. Extensions may be requested when needed, especially when there is no suitable reviewer for the paper in the system.
  3. Rebuttal and Discussion. Initially, reviews will only be accessible to the authors as they are submitted, but subsequently, they will become visible to all other parties including other reviewers. Authors have the option to post rebuttals and make revisions to their papers in light of the reviews. Additionally, reviewers and the AE may discuss the manuscript privately. Although authors are permitted to respond to reviews as soon as they are posted, it is advisable to wait until all reviews have been submitted before making any revisions to the PDF manuscript. This period is set to three weeks (two for Rebuttal one for Discussion periods), but extensions are possible if requested.
  4. Decision. Upon the discussion period, AE has one week to reach a decision for the paper, which is either an acceptance or a rejection. In the former case, the AE may request minor revisions required for the camera-ready submission. If a rejection occurs, the authors are notified of the decision.
  5. Withdrawal (if applicable). An author has the option to withdraw the submission prior to a decision being rendered. In such cases, they can initiate the withdrawal process themselves through OpenReview. The author will be required to provide a reason for the withdrawal, and although the submission will continue to be accessible on OpenReview, its status will be clearly indicated as "withdrawn."
  6. Post-decision period and publishing. The following lists the set of actions that need to be taken following the final decision.
    • Accept. Upon acceptance, authors are required to provide a "camera-ready" version incorporating the requested revisions if applicable. The camera-ready version should include a link to the paper's review page and must be verified by the AE for publication. Similarly, as in the initial submissions, DMLR welcomes the submission of any supplementary material to support the paper, such as datasets, Appendices, (audio)visual presentations, or code.
    • Reject. Authors whose submissions are rejected have the option to make revisions and submit their paper anew. However, this resubmission must be treated as an entirely new submission. They should include a link to the previously rejected submission and provide a description of the changes made since then. This process will be operated through OpenReview
    • Retract. Authors have the option to withdraw their submission even after it has been accepted. If an author chooses to retract their paper post-acceptance, they are required to submit a retraction statement through OpenReview. This statement will undergo review by the Editors-in-Chief, and all authors will be notified. Once the retraction is confirmed, the paper will remain accessible on OpenReview, but its status will be updated to indicate that it has been retracted. Additionally, the Editors-in-Chief hold the authority to withdraw a submission or retract an accepted paper if they identify unethical research, plagiarism (of any kind), or erroneous findings.
The expected turnaround time for the decision is usually between 4 and 6 months. However, it may take up to 10 months for exceptional cases.

Open access policy

DMLR ensures that all published material is promptly accessible without any associated costs. DMLR also does not require any fees or payments from authors, reviewers, action editors, or editors-in-chief.

Acknowledgments

DMLR procedures partially follow those of TMLR, JMLR, NeurIPS, and ICLR. Some of the sections above are adapted from those procedures.

Back to the top